Oracle Text Indexing performance in Unicode database
Forum folks,
I'm looking for overall performance thoughts in Text Indexing within a Unicode database. Part of our internal testing suites includes searching on values using contains filters over indexed binary and text documents. We've architected these tests such that they could be run in a suite or on their own, thus, the data is loaded at the beginning of each test and then the text indexes are created and populated prior to running any of the actual testing.
We have the same tests running on non-unicode instances of Oracle 11gR2 just fine, but when we run them against a Unicode instance, we are almost always seeing timing issues where the indexes haven't finished populating, thus our tests are saying we've only found n number of hits when we are expecting n+ 50 or in some cases n + 150 records to be returned.
We are just looking for some general information in regards to text indexing performance in a unicode database. Will we need to add sleep time to the testing to allow for the indexes to populate? How much time? We would rather not get into having to create different tests for unicode vs non-unicode, but perhaps that is necessary.
Any insight you could provide would be most appreciated.
Thanks in advance,
Dan
Roger,
Thanks much for your quick reply...
When you talk about Unicode, do you mean AL32UTF8?
--> Yes, this is the Unicode charset we are using.
Is the data the same in both cases, or are you indexing simple 7-bit ascii data in the one database, and foreign text (maybe Chinese?) in the UTF8 database?
With the same data, there should be virtually no difference in performance due to the AL32UTF8 database character set.
--> We have a data generation tool we utilize. For non-unicode data, we generate using all 256 characters in the ISO-8859-1 set. With our Unicode data for clobs, we generate using only the first 1,000 characters of UTF8 by setting up an array of code points...0 - 1000. For Blobs, we have sets of sample word documents and pdfs that are inserted, then indexed.
I'm not sure I understand your testing methodology. Do you run ( load-data, index-data, run-queries ) sequentially?
--> That is correct. We utilize the ctx_ddl package to populate the pending table and then to sync the index....The following is an example of the ddl we generate to create and populate the index:
create index "DBMEARSPARK_ORA80"."RESRESUMEDOC" on "DBMEARSPARK_ORA80"."RESUME" ("RESUMEDOC") indextype is CTXSYS.CONTEXT parameters(' nopopulate sync (every "SYSTIMESTAMP + INTERVAL ''30'' MINUTE" PARALLEL 2) filter ctxsys.auto_filter ') PARALLEL 2;
execute ctx_ddl.populate_pending('"DBMEARSPARK_ORA80"."RESRESUMEDOC"',null);
execute ctx_ddl.sync_index('"DBMEARSPARK_ORA80"."RESRESUMEDOC"',null,null,2);
If so, there should be no way that the indexes can be half-created. If not, don't you have some check to see if the index creation has finished before running the query test?
--> Excellent question....is there such a check? I have not found a way to do that yet...
Were you just lucky with the "non-unicode" tests that the indexing just happened to have always finished by the time you ran the queries?
--> This is quite possible as well. If there is a check to see if the index is ready, then we could add that into our infrastructure.
--> Thanks, again, for responding so quickly.
Edited by: djulson on Feb 12, 2013 7:13 AM
Similar Messages
-
Performance issue with Oracle Text index
Hi Experts,
We are on Oracle 11.2..0.3 on Solaris 10. I have implemented Oracle Text in our environment and I am facing a strange performance issue that is happening in our environment.
One sql having CONTAINS clause is taking forever - more than 20 minutes and still does not complete. This sql has a contains clause and an exists clause and a not exists clause.
Now if I remove the exists clause and a not exists clause , it completes fast. but with those two clauses it is just taking forever. It is late night so i am not able to post the table and sql query details and will do so tomorrow but based on this general description, are there any pointers for me to review?
sql query doing fine:
SELECT
U.CLNT_OID, U.USR_OID, S.MAILADDR
FROM
access_usr U
INNER JOIN access_sia S
ON S.USR_OID = U.USR_OID AND S.CLNT_OID = U.CLNT_OID
WHERE U.CLNT_OID = 'ABCX32S'
AND CONTAINS(LAST_NAME , 'TO%' ) >0
--sql query that hangs forever:
SELECT
U.CLNT_OID, U.USR_OID, S.MAILADDR
FROM
access_usr U
INNER JOIN access_sia S
ON S.USR_OID = U.USR_OID AND S.CLNT_OID = U.CLNT_OID
WHERE U.CLNT_OID = 'ABCX32S'
AND CONTAINS(LAST_NAME , 'TO%' ) >0
and exists (--one clause here wiht a few table joins)
and not exists (--one clause here wiht a few table joins);
--Now another strange thing I found is if instead of 'TO%' in this sql, if I were to use 'ZZ%' or 'L1%' it works fast but for 'TO%' it goes slow with those two exists not exists clauses!
I will be most thankful for the inputs.
OrauserNHi Barbara,
First of all, thanks a lot for reviewing the issue.
Unluckily making the change to empty_stoplist did not work out. I am today copying the entire sql here that has this issue and will be most thankful for more insights/pointers on what can be done.
Here is the entire sql:
SELECT U.CLNT_OID,
U.USR_OID,
S.EMAILADDRESS,
U.FIRST_NAME,
U.LAST_NAME,
S.JOBCODE,
S.LOCATION,
S.DEPARTMENT,
S.ASSOCIATEID,
S.ENTERPRISECOMPANYCODE,
S.EMPLOYEEID,
S.PAYGROUP,
S.PRODUCTLOCALE
FROM ACCESS_USR U
INNER JOIN
ACCESS_SIA S
ON S.USR_OID = U.USR_OID AND S.CLNT_OID = U.CLNT_OID
WHERE U.CLNT_OID = 'G39NY3D25942TXDA'
AND EXISTS
(SELECT 1
FROM ACCESS_USR_GROUP_XREF UGX
INNER JOIN ACCESS_GROUP RELG
ON RELG.CLNT_OID = UGX.CLNT_OID
AND RELG.GROUP_OID = UGX.GROUP_OID
INNER JOIN ACCESS_GROUP G
ON G.CLNT_OID = RELG.CLNT_OID
AND G.GROUP_TYPE_OID = RELG.GROUP_TYPE_OID
WHERE UGX.CLNT_OID = U.CLNT_OID
AND UGX.USR_OID = U.USR_OID
AND G.GROUP_OID = 920512943
AND UGX.INCLUDED = 1)
AND NOT EXISTS
(SELECT 1
FROM ACCESS_USR_GROUP_XREF UGX
INNER JOIN
ACCESS_GROUP G
ON G.CLNT_OID = UGX.CLNT_OID
AND G.GROUP_OID = UGX.GROUP_OID
WHERE UGX.CLNT_OID = U.CLNT_OID
AND UGX.USR_OID = U.USR_OID
AND G.GROUP_OID = 920512943
AND UGX.INCLUDED = 1)
AND CONTAINS (U.LAST_NAME, 'Bon%') > 0;
Like I said before if the EXISTS and NOT EXISTS clause are removed it works in sub-second. But with those EXISTS and NOT EXISTS CLAUSE IT TAKES ANY WHERE FROM 25 minutes to more than one hour.
NOte also that it was not TO% but Bon% in the CONTAINS clause that is giving the issue - sorry that was wrong on my part.
Also please see below the ORACLE TEXT index defined on the table ACCESS_USER:
--definition of preferences used in the index:
SET SERVEROUTPUT ON size unlimited
WHENEVER SQLERROR EXIT SQL.SQLCODE
DECLARE
v_err VARCHAR2 (1000);
v_sqlcode NUMBER;
v_count NUMBER;
BEGIN
ctxsys.ctx_ddl.create_preference ('cust_lexer', 'BASIC_LEXER');
ctxsys.ctx_ddl.set_attribute ('cust_lexer', 'base_letter', 'YES'); -- removes diacritics
EXCEPTION
WHEN OTHERS
THEN
v_err := SQLERRM;
v_sqlcode := SQLCODE;
v_count := INSTR (v_err, 'DRG-10701');
IF v_count > 0
THEN
DBMS_OUTPUT.put_line (
'The required preference named CUST_LEXER with BASIC LEXER is already set up');
ELSE
RAISE;
END IF;
END;
DECLARE
v_err VARCHAR2 (1000);
v_sqlcode NUMBER;
v_count NUMBER;
BEGIN
ctxsys.ctx_ddl.create_preference ('cust_wl', 'BASIC_WORDLIST');
ctxsys.ctx_ddl.set_attribute ('cust_wl', 'SUBSTRING_INDEX', 'true'); -- to improve performance
EXCEPTION
WHEN OTHERS
THEN
v_err := SQLERRM;
v_sqlcode := SQLCODE;
v_count := INSTR (v_err, 'DRG-10701');
IF v_count > 0
THEN
DBMS_OUTPUT.put_line (
'The required preference named CUST_WL with BASIC WORDLIST is already set up');
ELSE
RAISE;
END IF;
END;
--now below is the code of the index:
CREATE INDEX ACCESS_USR_IDX3 ON ACCESS_USR
(FIRST_NAME)
INDEXTYPE IS CTXSYS.CONTEXT
PARAMETERS('LEXER cust_lexer WORDLIST cust_wl SYNC (ON COMMIT)');
CREATE INDEX ACCESS_USR_IDX4 ON ACCESS_USR
(LAST_NAME)
INDEXTYPE IS CTXSYS.CONTEXT
PARAMETERS('LEXER cust_lexer WORDLIST cust_wl SYNC (ON COMMIT)');
The strange thing is that, like I said, If I remove the exists clause the query returns very fast. Also if I modify the query to use only one NOT EXISTS clause and remove the other EXISTS clause it returns in less than one second. Also if I remove the EXISTS clause and use only the NOT EXISTS clause it returns in less than 4 seconds. But with both clauses it runs forever!
When I tried to get dbms_xplan.display_cursor to get the query plan (for the case of both exists and not exists clause in the query), it said that previous statement's sql id was 0 or something like that so that I was not able to see the query plan. I will keep trying to get this plan (it takes 25 minutes to one hour each time but will get this info soon). Again any pointers are most helpful.
Regards
OrauserN -
About index memory parameter for Oracle text indexes
Hi Experts,
I am on Oracle 11.2.0.3 on Linux and have implemented Oracle Text. I am not an expert in this subject and need help about one issue. I created Oracle Text indexes with default setting. However in an oracle white paper I read that the default setting may not be right. Here is the excerpt from the white paper by Roger Ford:
URL:http://www.oracle.com/technetwork/database/enterprise-edition/index-maintenance-089308.html
"(Part of this white paper below....)
Index Memory As mentioned above, cached $I entries are flushed to disk each time the indexing memory is exhausted. The default index memory at installation is a mere 12MB, which is very low. Users can specify up to 50MB at index creation time, but this is still pretty low.
This would be done by a CREATE INDEX statement something like:
CREATE INDEX myindex ON mytable(mycol) INDEXTYPE IS ctxsys.context PARAMETERS ('index memory 50M');
Allow index memory settings above 50MB, the CTXSYS user must first increase the value of the MAX_INDEX_MEMORY parameter, like this:
begin ctx_adm.set_parameter('max_index_memory', '500M'); end;
The setting for index memory should never be so high as to cause paging, as this will have a serious effect on indexing speed. On smaller dedicated systems, it is sometimes advantageous to temporarily decrease the amount of memory consumed by the Oracle SGA (for example by decreasing DB_CACHE_SIZE and/or SHARED_POOL_SIZE) during the index creation process. Once the index has been created, the SGA size can be increased again to improve query performance."
(End here from the white paper excerpt)
My question is:
1) To apply this procedure (ctx_adm.set_parameter) required me to login as CTXSYS user. Is that right? or can it be avoided and be done from the application schema? This user CTXSYS is locked by default and I had to unlock it. Is that ok to do in production?
2) What is the value that I should use for the max_index_memory should it be 500 mb - my SGA is 2 GB in Dev/ QA and 3GB in production. Also in the index creation what is the value I should set for index memory parameter - I had left that at default but how should I change now? Should it be 50MB as shown in example above?
3) The white paper also refer to rebuilding an index at some interval like once in a month: ALTER INDEX DR$index_name$X REBUILD ONLINE;
--Is this correct advice? i would like to ask the experts once before doing that. We are on Oracle 11g and the white paper was written in 2003.
Basically while I read the paper, I am still not very clear on several aspects and need help to understand this.
Thanks,
OrauserNPerhaps it's time I updated that paper
1. To change max_index_memory you must be a DBA user OR ctxsys. As you say, the ctxsys account is locked by default. It's usually easiest to log in as a DBA and run something like
exec ctxsys.ctx_adm.set_parameter('MAX_INDEX_MEMORY', '10G')
2. Index memory is allocated from PGA memory, not SGA memory. So the size of SGA is not relevant. If you use too high a setting your index build may fail with an error saying you have exceeded PGA_AGGREGATE_LIMIT. Of course, you can increase that parameter if necessary. Also be aware that when indexing in parallel, each parallel process will allocated up to the index memory setting.
What should it be set to? It's really a "safety" setting to prevent users grabbing too much machine memory when creating indexes. If you don't have ad-hoc users, then just set it as high as you need. In 10.1 it was limited to just under 500M, in 10.2 you can set it to any value.
The actual amount of memory used is not governed by this parameter, but by the MEMORY setting in the parameters clause of the CREATE INDEX statement. eg:
create index fooindex on foo(bar) indextype is ctxsys.context parameters ('memory 1G')
What's a good number to use for memory? Somewhere in the region of 100M to 200M is usually good.
3. No - that's out of date. To optimize your index use CTX_DDL.OPTIMIZE_INDEX. You can do that in FULL mode daily or weekly, and REBUILD mode perhaps once a month. -
Problem with oracle text indexes during import
We have a 9.2.0.6 database using oracle text features on a server with windows 2000 5.00.2195 SP4.
We need to export its data ( user ARIANE only ) and then import the result into another 9.2.0.6 database.
The import never comes to an end.
The only way to make it work is to use the "indexes=n" clause.
Then ( without the indexes ), we tried to create manually the oracle text indexes.
We get this error :
CREATE INDEX ARIANE.DOSTEXTE_DTTEXTE_CTXIDX ON ARIANE.DOSTEXTE (DTTEXTE)
INDEXTYPE IS CTXSYS.CONTEXT PARAMETERS('lexer ariane_lexer stoplist ctxsys.default_stoplist storage ariane_storage');
ORA-29855: erreur d'exécution de la routine ODCIINDEXCREATE
ORA-20000: Erreur Oracle Text :
DRG-10700: préférence inexistante : ariane_lexer
ORA-06512: à "CTXSYS.DRUE", ligne 157
ORA-06512: à "CTXSYS.TEXTINDEXMETHODS", ligne 219
We then tried to uninstall Oracle text and install it ( My Oracle Support [ID 275689.1] ). The index creation above still fails.
We also checked our Text installation and setup through My Oracle Support FAQ ( ID 153264.1 ) and everything seems ok.
Do we have to create some ARIANE* lexer preferences through specific pl/sql ( ctx_report* ? ) before importing anything from the ARIANE user ?
What do we need to do exactly when exporting data with oracle text features from one database to another given we used to restore the database through a copy of the entire windows files ?
Is there a specific order to follow to succeed an import ?
Thank you for your help.
Jean-michel, Nemours, FRANCEHi
index preferences are not exported, ie ariane_lexer + ariane_storage, only the Text index metada, thus the DRG-10700 from index DDL on target/import DB.
I recommend to use ctx_report.create_index_script on source/export DB, see Doc ID 189819.1 for details, export with indexes=N and then create text indexes manually after data import.
-Edwin -
Oracle Text indexation problem
Hello all,
we recently migrate one of our database from oracle 10.2.0.2 to an Oracle 10.2.0.4, and since then, Oracle Text indexer engine is unable to work properly.
Indexing datas systematically result in the following error :
DRG-50857: oracle error in dreii0fsh
ORA-06550: line 1, column 106:
PLS-00302: component 'S_MODE' must be declared
ORA-06550: line 1, column 10:
PL/SQL: Statement ignored
Note that datas that were indexed with the previous version of Oracle can still be found with "Select... where contains...", though new datas can't be indexed at all.
Can anybody help me with this issue ?
Edited by: methos on 11 mai 2010 06:59See on metalink:
'DRG-50857: oracle error in dreii0fsh' when creating TEXT index [ID 750898.1]
It seems you upgraded the binaries,but not Oracle Text itself.
Werner -
Hi,
I created an oracle text index on 10G database on a fairly large table (59 Mil recs) but when i go to Analyze it, i get the ora-29864 error message 'analyzing domain indexes marked LOADING/FAILED not supported'. The status of the index is valid. The index was created well over 24 hours ago. is it perhaps because the index is still loading?
Has anyone run across this problem?
Thanks"ORA-29864: analyzing domain indexes marked LOADING/FAILED not supported
Cause: Tried to analyze a domain index which was marked as LOADING or FAILED.
Action: If the index was marked LOADING, wait till it is marked valid before retrying the analyze. If index was marked FAILED, drop it OR rebuild it and retry the analyze." -
Oracle Text Index Error - DRG-10700: preference does not exist: CTXSYS.DEFA
Hi we are having a Oracle 11g db.And we have installed CTX SYS package and provided the grants to the required schema.
But while creating the oracle text index, following error is thrown:
ORA-29855: error occurred in the execution of ODCIINDEXCREATE routine
ORA-20000: Oracle Text error:
DRG-10700: preference does not exist: CTXSYS.DEFAULT_LEXER
ORA-06512: at "CTXSYS.DRUE", line 160
ORA-06512: at "CTXSYS.TEXTINDEXMETHODS", line 365
Following grnats are given :
GRANT EXECUTE ON CTXSYS.CTX_DDL TO <schema_name>;
GRANT APPOWN TO <schema_name>;
GRANT CTXAPP TO <schema_name>;
I have verifyed the ctxsys.CTX_PREFERENCES view which is having an entry as :
SELECT PRE_OWNER, PRE_NAME
FROM CTXSYS.CTX_PREFERENCES
WHERE PRE_NAME = 'DEFAULT_LEXER';
The Result is :
PRE_OWNER PRE_NAME
SYS DEFAULT_LEXER
Can you please assist if there is a way to update the permission on this DEFAULT_LEXER preference to set the PRE_OWNER as CTXSYS ?
ThanksThat's odd. If I run the same query I see:
SQL> SELECT PRE_OWNER, PRE_NAME
2 FROM CTXSYS.CTX_PREFERENCES
3 WHERE PRE_NAME = 'DEFAULT_LEXER';
PRE_OWNER PRE_NAME
CTXSYS DEFAULT_LEXERAre all your defaults like thsi? What do you see if you run
select pre_owner, pre_name from ctxsys.ctx_preferences where pre_name like 'DEFAULT%';
It would be easy enough to update the underlying table DR$PREFERENCES to fix the owner, but who knows what else might be wrong.
Has this database been updated from a 9i installation by any chance? -
Error When Creating Oracle Text index using Lexer Keyword
Hi All,
I am getting following error when i creating oracle text index using lexer & stoplist keyword.
Pls Help me if any body know.
Thanks in Advance.
Error starting at line 1 in command:
CREATE INDEX TXT_INX_TEXT_SEARCH ON TEXT_SEARCH (BFILE_DOC)
Post INDEXTYPE IS "CTXSYS"."CONTEXT" LOCAL (
PARTITION "BEFORE_2007" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)') ,
PARTITION "Q1_2007" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "Q2_2007" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "Q3_2007" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "Q4_2007" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "Q1_2008" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "Q2_2008" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "Q3_2008" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "Q4_2008" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "Q1_2009" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "Q2_2009" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "Q3_2009" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "Q4_2009" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "THE_REST" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)')
Error at Command Line:1 Column:13
Error report:
SQL Error: ORA-29855: error occurred in the execution of ODCIINDEXCREATE routine
ORA-20000: Oracle Text error:
DRG-11000: invalid keyword LEXER
ORA-06512: at "CTXSYS.DRUE", line 160
ORA-06512: at "CTXSYS.TEXTINDEXMETHODS", line 365
29855. 00000 - "error occurred in the execution of ODCIINDEXCREATE routine"
*Cause: Failed to successfully execute the ODCIIndexCreate routine.
*Action: Check to see if the routine has been coded correctly.
Regards,
Jack R.Hi,
it works if you put an extra PARAMETERS clause at the end so the creation looks like:
CREATE INDEX TXT_INX_TEXT_SEARCH ON TEXT_SEARCH (BFILE_DOC)
INDEXTYPE IS "CTXSYS"."CONTEXT" LOCAL (
PARTITION "BEFORE_2007" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)') ,
PARTITION "Q1_2007" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "Q2_2007" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "Q3_2007" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "Q4_2007" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "Q1_2008" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "Q2_2008" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "Q3_2008" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "Q4_2008" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "Q1_2009" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "Q2_2009" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "Q3_2009" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "Q4_2009" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)'),
PARTITION "THE_REST" PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)')
PARAMETERS ('LEXER dd_lexer STOPLIST dd_stoplist SYNC (ON COMMIT)') <== Added
Hope this helps
Herald ten Dam -
Oracle text indexed view is possible
Oracle text indexed view is possible???
ok,
My table name is T_DOC :
ID----------------> NUMBER(30)
DESCRIPTION-------> VARCHAR2(2000 BYTE)
DOC---------------> BLOB
FILENAME----------> VARCHAR2(2000 BYTE)
MIMETYPE----------> VARCHAR2(2000 BYTE)
LAST_UPDATE_DATE--> DATE
T_DOC
| Id | DESCRIPTION | DOC | FILENAME | MIMETYPE | LAST_UPDATE_DATE |
| 1 | THE DOG | *(!BLOB) | THE_CAT.PDF | application/pdf | 20/05/2010 15:06:15 |
| 2 | THE BIRD | **(!BLOB) | THE_BIRD.PDF | application/pdf | 20/05/2010 15:06:15 |
| 3 | THE HUMAN AND CAT | ***(!BLOB) | THE_HUMAN.PDF | application/pdf | 20/05/2010 15:06:15 |
* is a document .pdf with content: "the dog and cat"
** is a document .pdf with content: "the bird in house"
*** is a document .pdf with content: "the human from USA"
Index the columns DESCRIPTION, DOC (document content), FILENAME
begin
ctx_ddl.create_preference('idxDoc_lx', 'BASIC_LEXER');
ctx_ddl.set_attribute (' idxDoc_lx ', 'MIXED_CASE', 'NO');
end;
begin
ctx_ddl.create_preference('idxDoc_ds', 'MULTI_COLUMN_DATASTORE');
ctx_ddl.set_attribute ('idxDoc_ds', 'COLUMNS', 'DOC, FILENAME, DESCRIPTION');
end;
CREATE INDEX IDX_DOC
ON T_DOC (FILENAME)
INDEXTYPE IS CTXSYS.CONTEXT
PARAMETERS ('lexer idxDoc_lx
datastore idxDoc_ds
filter CTXSYS.AUTO_FILTER
sync (on commit)');Search Query:
select ID
from T_DOC
where CONTAINS (DOCUMENTO, 'CAT', 1) > 0 RESULT ID = 1
WHY NOT ALSO Returned ID 3 ?????? -
How to compute a global SCORE over a few oracle text indexed tables?
Dear experts!
I want to search a website with Oracle Text. The website consists of four tables:
- site
- chapter
- text
- binaries
Each table has two or three columns which should be indexed with oracle text. So I have created a MULTI_COLUMN_DATASTORE oracle text index on each table - So I have four indexes on my website.
When I want to search over the website I have to join my 4 tables (4 contain clauses). So how do I get a global SCORE over these 4 contains clauses?
The next question is can I change the weight of my text indexes (useful for the search hit list)? For example the highest weight has the site index, the second highest weight the chapter index and so on?
Thanks
MarkusIf it's a simple JOIN, then you could just add the scores for each CONTAINS clause
select score(1)+score(2)+score(3)+score(4)
from table1 t1,table2 t2, table3 t3,table4 t4
where [join conditions]
and contains(t1.col, 'xxx', 1) > 0 or
contains(t2, col, 'xxx', 2) > 0 or
... etc
then to change the weight you just add a multiplying factor.
Can't help thinking it's probably more complex than this, though. -
Why oracle text index column taking long time
why oracle text index column is taking long time to return result.I created text index on a column if I run the query on a single table result is very fast.If I join table with other table (10 records only )
it is taking long time but in explain plan it is searching by index only.
I created this index for searching a varchar2 column,the data is comma seperated values like ( UK,US,IT,BR) and the table having records 20 lakhs.Normally if I query with like operater
( like '%US%' ) it is taking full table scan because I am using '%' both sides. Please help me on this regard how to search the data with less time. Here is may sample code and explain plan.
SQL*Plus: Release 9.2.0.1.0 - Production on Wed Jan 28 16:54:22 2009
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.
Connected to:
Oracle9i Enterprise Edition Release 9.2.0.1.0 - Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.1.0 - Production
SQL> set timing on
SQL> set linesize 180
SQL> explain plan for SELECT T.esongid FROM (SELECT A.ESONGID FROM wcmedeco.EDECO_ESONGS_TERR_CTRY
A WHERE CONTAINS(A.TERR_CTRY_NAMES,'US')>0
2 GROUP BY A.ESONGID)K,T
3 WHERE K.ESONGID=T.ESONGID;
Explained.
Elapsed: 00:00:00.01
SQL> select *from table(dbms_xplan.display);
PLAN_TABLE_OUTPUT
| Id | Operation | Name | Rows | Bytes | Cost |
| 0 | SELECT STATEMENT | | 1 | 26 | 4 |
| 1 | NESTED LOOPS | | 1 | 26 | 4 |
| 2 | VIEW | | 1 | 13 | 4 |
| 3 | SORT GROUP BY | | 1 | 89 | 4 |
| 4 | TABLE ACCESS BY INDEX ROWID| EDECO_ESONGS_TERR_CTRY | 1 | 89 | 2 |
| 5 | DOMAIN INDEX | IDX_TERR_CTRY_NAMES | | | 0 |
| 6 | INDEX RANGE SCAN | IDX_ESONGID_T | 1 | 13 | 1 |
PLAN_TABLE_OUTPUT
Note: cpu costing is off, 'PLAN_TABLE' is old version
14 rows selected.
Elapsed: 00:00:00.00
SQL> Regards,
RajasekharYou have not formatted your code properly so we cannot see the query you're executing. Please put some line breaks in.
Secondly, how fresh are the statistics on those tables? Are you really returning one record out of twenty million?
Cheers, APC
blog: http://radiofreetooting.blogspot.com -
Hello
Had a query on oracle Text Index.
1. I have created a table with column ID as sequence & column description as Varchar2(500)
2. Inserted 3 records
3. I created the Oracle Text index on column "description" as INDEXTYPE IS CTXSYS.CONTEXT
4. Run a select using "contains" , get the records as required.
5. I insert another new record & commit;
6. Run the same old query using contains but for the new records I get 0 records
7. Drop the Index and Recreate the index using same old index on column "description" as INDEXTYPE IS CTXSYS.CONTEXT
8. Run the same old query using contains but for the new records and now I get the records as required.
Does that mean every time I insert new records I have to rebuild the Oracle Text Index ?
Your Help will be much appreciated
Regards
HarshadYou need to use some method of updating your index. You can drop and recreate or you can rebuild or you can use ctx_ddl.sync_index or you can include sync(on commit) in your parameters when you create the index or various other methods of periodic synchronization and optimization.
-
SMB Server using Oracle Text Index
Hi folks,
I am looking into IFS and can't find any info
on whether IFS is able to use the Oracle Text
indexing, when accessed through SMB (directory mount).
If I search for a file containing a given string
is that search executed within IFS and only the
resultset returned to the client, or is every
file read from IFS delivered to the client,
searched there and then thrown away if it doesn't
contain the string (rendering the Text Index useless)?
Does anybody have background info / experience with this?
Regards
Jan-PeterHi folks,
I am looking into IFS and can't find any info
on whether IFS is able to use the Oracle Text
indexing, when accessed through SMB (directory mount).
If I search for a file containing a given string
is that search executed within IFS and only the
resultset returned to the client, or is every
file read from IFS delivered to the client,
searched there and then thrown away if it doesn't
contain the string (rendering the Text Index useless)?
Does anybody have background info / experience with this?
Regards
Jan-Peter To answer your question, it would help to have a little more context (pardon the pun). Are you trying to (1) understand how the 9iFS Windows UI uses Oracle Text, or (2) are you trying to write a client application that somehow uses the Text index on 9iFS content? -
Oracle Text index hangs(Oracle 11g linux enterprise edition)
Hi guru,
One very criticial and showstopper issue coming in Oracle Text Indexing.
I have table ResourceTable(ResId,Contents blob ,docformat)
I am creating text index on Contents column by using follwing command:
create index [IndexName] on [TableName] (contents) indextype is ctxsys.context ('lexer mylex stoplist ctxsys.default_stoplist format column ISDOCFORMAT sync(every "sysdate+1/24") storage my_text_storage memory 200M')parallel 2
I found there is one document of text/html type which hangs oracle text indexing thread.Oracle doest not give any error message keep using CPU 100% and no progress.
I then create test table with same stucture and put this corrupted documet record in this table.and did indexing again but its not doing index /not ignoring it and hangs .We have also tried to use timeout feature of inso filter so that if such type of document comes at the time of indexing oracle text indexing process will bypass it after specific time interval but still same issue comes and oracle is not bypassing such faulty document.
This is realy show stopper issue and any kind of help will be greatly appricited.You need to raise an SR with Oracle Support for this, and send them the file which is causing the hang. It should then be easy to investigate the problem and schedule a fix.
If you don't have a support contract, you could send the file to me and I'll raise a bug. However, if we fix it you won't be able to get the fix until it appears in the next downloadable version.
- Roger
roger.ford @ oracle.com -
Help with creating oracle text index on 2 columns with partial html data
Hi,
I need to create an oracle text index on 2 columns.
TITLE - varchar(255) = contains plain text data
DESCRIPTION - CLOB = contains partial HTML data
This is what I created.
begin
ctx_ddl.create_preference ('Title_Description_Pref', 'MULTI_COLUMN_DATASTORE');
ctx_ddl.set_attribute('Title_Description_Pref', 'columns', 'TITLE, DESCRIPTION');
end;
begin
ctx_ddl.create_preference ('bid_lexer', 'BASIC_LEXER');
ctx_ddl.set_attribute('bid_lexer', 'index_stems', 'ENGLISH');
ctx_ddl.create_section_group('htmgroup', 'HTML_SECTION_GROUP');
end;
create index Bid_Title_Index on Bid(title) indextype is ctxsys.context parameters ('LEXER bid_lexer sync (every "sysdate+(1/24)")');
create index Bid_Title_Desc_Index on Bid(description) indextype is ctxsys.context parameters ('LEXER bid_lexer DATASTORE Title_Description_Pref sync (every "sysdate+(1/24)") filter ctxsys.null_filter section group htmgroup');
The problem is when I do a CONTAINS(description, '$(auction)')>0. I get results where the descriptions have the "auction" word (which is correct). But, the results also returned rows where the search word is inside an IMG tag. e.g. <img src="http://auction.de/120483" alt="Auction Logo"/>.
What I would like is to exclude rows where the search word is inside HTML tag attributes, results expected are rows having <a>Auction</a> or <p>For Auction</p> ... etc. Basically stripping the html tags and leave the text contents.
I'd appreciate some input.
Thanks,
AmielHi,
I need to create an oracle text index on 2 columns.
TITLE - varchar(255) = contains plain text data
DESCRIPTION - CLOB = contains partial HTML data
This is what I created.
begin
ctx_ddl.create_preference ('Title_Description_Pref', 'MULTI_COLUMN_DATASTORE');
ctx_ddl.set_attribute('Title_Description_Pref', 'columns', 'TITLE, DESCRIPTION');
end;
begin
ctx_ddl.create_preference ('bid_lexer', 'BASIC_LEXER');
ctx_ddl.set_attribute('bid_lexer', 'index_stems', 'ENGLISH');
ctx_ddl.create_section_group('htmgroup', 'HTML_SECTION_GROUP');
end;
create index Bid_Title_Index on Bid(title) indextype is ctxsys.context parameters ('LEXER bid_lexer sync (every "sysdate+(1/24)")');
create index Bid_Title_Desc_Index on Bid(description) indextype is ctxsys.context parameters ('LEXER bid_lexer DATASTORE Title_Description_Pref sync (every "sysdate+(1/24)") filter ctxsys.null_filter section group htmgroup');
The problem is when I do a CONTAINS(description, '$(auction)')>0. I get results where the descriptions have the "auction" word (which is correct). But, the results also returned rows where the search word is inside an IMG tag. e.g. <img src="http://auction.de/120483" alt="Auction Logo"/>.
What I would like is to exclude rows where the search word is inside HTML tag attributes, results expected are rows having <a>Auction</a> or <p>For Auction</p> ... etc. Basically stripping the html tags and leave the text contents.
I'd appreciate some input.
Thanks,
Amiel
Maybe you are looking for
-
I want to update my payment information but it keeps saying my security code is invalid and my payment method was declined. However, I got a short message from my bank saying the charge by Apple was successful? How could it be if the payment informa
-
Allow Software to Delete Channels, but Not User
I am using the OnDeleteRequest action to prevent the user from deleting channels in my custom device. However I realized that this also prevents code I write (in this case executing on a right-click action) from deleting channels. I need a way to d
-
Who wants to be a Millionaire ??? CODE???
hi i'm tryin to make a basic game who wants to be a millionaire...i got the following..layout of the textfield textarea etc.. but i cant seem to use the vectors to store the questions from a txt file then transfer it over to the textarea box or radio
-
I-Phone not syncing correct time zone in ical, but computer time stays the same
Lately when I put in appointments in my calendar on my Iphone 4S either directly,or in my mac, the time zone drops back 3 hours in just the Iphone only. The correct time is still on my calendar in my mac. I checked my Iphone settings to make sure th
-
Transfer SLD content between SLDs in SLD 640
Is it possible to transfer content between two SLD systems in SLD version 640? 1.) I have a SLD (call it sldA) that contains the ABAP and JAVA data about DEV and QA system. 2.) I have another SLD (call it sldB) that contains the ABAP and JAVA data a