Full-text search containstable AND logical operator performance problem

I have the following clause in my SQL statement:
CONTAINSTABLE(subject_ifts, SearchText, '"smith*" AND "n*"', LANGUAGE 1033)
It takes approximately 60 seconds to return 840 rows from a total of 4 millions rows in the searched table. If the search condition is changed to '"smith*"' it returns 840 rows in less than 1 second (ie. all rows containing the text "smith"
also contain the text "n"). It seems that the search for "n*" takes a long time to return rows as almost all 4 million rows contain this text. (Note: The search criteria is passed as a parameter into a stored procedure at runtime based
on the search criteria input by the user in the UI).
Is there any way to make SQL Server perform its search for the text "n*" just on the resultset of the 840 rows returns from the "smith*" search? Theoretically this should return rows a lot quicker basing the search on 840 rows
rather than 4 million. However, I cannot seem to implement this effectively. I have tried using CTEs and JOIN to no avail. Any help greatly appreciated.
Graham Goodwin Email: [email protected]

Hello,
I am trying to involve someone more familiar with this topic for a further look at this issue. Sometime delay might be expected from the job transferring. Your patience is greatly appreciated.
Thank you for your understanding and support.
Regards,
Fanny Liu
If you have any feedback on our support, please click
here.
Fanny Liu
TechNet Community Support

Similar Messages

Is it possible to ignore noise words conditionally in working with Full text search containstable

I have a question on stoplist file. I need to search for exact phrase string("this is the incident") which contains noise words. As part of the FT search engine, during parsing it eliminates noise word and search on remaining string in
the given phrase.
let us say there are 10 rows which contains the term "incident" in the FT table . and 1 row which has the exact phrase.i.e."this is the incident".
if we use containstable() to search for "this is the incident", we are getting 10 rows instead of 1 row.
To resolve the issue, we have 3 solutions
1.either stoplist file needs to be modified to remove the words (this,is,the)
2. set stoplist = OFF.
3.empty stoplist.
Apart from the above solutions, is there any better solution with out touching noise words file list.
If any solution that provides flexibility to ignore noise words conditionally at one time and not to ignore them.
Please provide your suggestion.
kkprasad

One question that I ask is: Why would I want to exclude noise words?
Noise words were created to limit the size of the full text indexes and avoid processing the many 'this', 'is', and 'the' common words. But the disadvantage of doing so is that you cannot find some things as you would like.
My feeling is that computers are more powerful and have more storage and it is often better to just index everything. As long as your search does not include 'the', then the large number of 'the's in the system will pretty much be ignored.
NOTE: If you change the noise words, including SET STOPLIST = OFF, you have to rebuild the index in order for it to implement your decision.
Of course, for very, very large full text indexes you would need to test.
Is your full text search on relation database columns, e.g. Description NVARCHAR(1000) or are you searching Word, Excel, and other more complex data?
If your full text is relational columns, it might be that you could:
1. Select only the fulltextkey into a temp table (e.g. #FTSfulltextkey) from the full text index using noise words. That would give you 10 rows.
2. Then directly query the table to find the string as you define above. (But remember that punctuation and symbols are generally ignored by Full Text Indexing, but would still be there in the string of text.)
SELECT *
FROM MyTextTable T
JOIN #FTSfulltextkey K
ON T.fulltextkey = k.fulltextkey
WHERE T.Description like '%this is the incident%'
Full text search is powerful, but it has limits. And the behaviour changes depending on the Language of the search.
RLF

Full text search in Filenet via SAP DMS

Hello experts,
Hopefully you can help me with the following. My customer has SAP DMS and uses FileNet (v4.0) as content repository. They use Verity as tool for full text search (SAP and non-SAP). Verity has it's own user interface / search screen. Question is if it is possible for SAP (easy)DMS users to do full text search (via CV04N or EasyDMS searchscreen) and search in the FileNet repository (directly with TREX or via Verity). The customer has the ACSAP connector up and running.
I've implemented TREX with SAP content server and index server before, but I'm not familiar with FileNet possibilities in that area, so hopefully someone has experience with this?
thanks in advance, Erwin

Dear NAG
For text search you have activate TREX Server
follow the steps
Set TREX
A:- Transaction: SRMO
Install TREX , and create RFC destination to connect to
TREX.
B:- Transaction: SKPR06
Check the flag to Document Area "DMS" to utilize this document area for
document search.
For Indexing
A: Transaction: SE38
Run the report program "RSTIRIDX_REINDEX" to create Index.
Also, run the report program "RSTIRIDX" to create Index.
Set mime type for the text search
A: SPRO --
Cross-Application Components > Document Management System > General
Data > Settings for Storage Systems > Create MIME types for full
text search,
Enter MIME type like
"application/msword",
"application/pdf".
Retrive document via Test Search
A: Then, test retrieval document search using
Transaction: SKPR07.
Enter Document Class "DMS_PCD1", Language "EN", Document Class "DMS",
and try "TEST search".
Regards
Tushar Dave

AIR and full-text search in SQLite

Hi,
can I use full-text indexing capabilities of SQLite with my
AIR app?
I've created a virtual table in database using fts3 module
(via command line of sqlite.exe). But when I try to insert a row
into this table, I get an error message like
...SQLError: 'Error #3115: SQL Error.', details:'no such
module: fts3', operation:'execute', detailID:'2501'...
Does AIR Data library support FTS in SQLite?

Yes, Adobe has disabled loadable modules but that does not
prevent them from including FTS3 in AIR's SQLite.
PLEASE, statically compile in both the FTS3 module and the
R*Tree Spatial Index module.
The beauty of AIR is that you can have an application that
works offline (duh!, right?). When it's online, you can do a full
text search in it, using the powerful server-side database's
functionality. Now when you go offline, you're completely crippled!
... Because Adobe didn't put FTS3 in SQLite.
' Same with Spatial functionality.
I'd sure like to know how companies make these decisions.
>sigh< It is EXTREMELY frustrating to be limited by
apparently arbitrary decisions like these. I sure hope this was not
motivated by a desire to "encourage" people to use their commercial
productions such as BlazeDS. I'd like to think better of Adobe than
that.
Again, PLEASE include these modules in the next release.
Scott

Problems using and configuring Oracle 10gR2 database full-text search

I am having problems trying to set up full-text indexing and search with Universal Content Management (UCM). I followed the Oracle Content Server Installation Guide for windows at [http://download-west.oracle.com/docs/cd/E10316_01/cs/cs_doc_10/documentation/integrator/install_cserver_win_10en.pdf].
What I did was:
1. Modify E:\oracle\ucm\server\config\config.cfg by adding SearchIndexerEngineName=DATABASE.FULLTEXT to the end of the file.
2. Restart the content server.
3. Rebuild the search indexing using Repository Manager.
However, I keep seeing the following error when I query by entering words in the "Full-Text Search" box.
Unable to retrieve search results. Unable to retrieve search results. Unable to create result set for query 'SELECT IdcColl1.dID, dDocName, dDocTitle, dDocType, dRevisionID, dSecurityGroup, dDocAuthor, dDocAccount, dRevLabel, dFormat, dOriginalName, dExtension, dWebExtension, dInDate, dOutDate, dCreateDate, dPublishType, dRendition1, dRendition2, VaultFileSize, WebFileSize, URL, dFullTextFormat, dFullTextCharset, DocMeta.*
FROM IdcColl1, DocMeta
WHERE IdcColl1.dID=DocMeta.dID AND (((CONTAINS(dDocFullText,'test') > 0 ))) ORDER BY dInDate Desc'. ORA-20000: Oracle Text error:
DRG-10599: column is not indexed
Some web searches suggested the following (all of which I have tried but not resolved this problem).
1. Publish the schema using Configuration Manager (applet) and then rebuild index
2. Set the dDocFullText as a "zone field". This is not possible, because dDocFullText does not show up under the list of fields under "Database" or "DatabaseFullText" for the Search Engine drop down (when using Zone Fields Configuration).
3. Reboot the server (did not work either).
I logged onto the Oracle database and checked the IdcColl1 table. There is indeed, no index for the field, dDocFullText. There is only 1 index for the field, did. The field, dDocFullText, is a BLOB. The question is, if I am supposed to create an index manually for this field, how would I do it? A web search has not been fruitful in answering this question.
Here are my server settings.
For UCM:
Operating System: Windows 2003 Enterprise
UCM : 10gR3
Memory: 1 GB
Web Server: Apache 2.2.11
For Oracle:
Operating System: Windows 2003 Enterprise
Oracle: 10gR2
Memory: 1 GB
Thanks.

I found out what the problem was. The problem was that I had to create the role, stellent_role, as described in the installation manual. After I created this role and assigned the database user to this role, a restart of the Content Server services and collection rebuild of the index fixed the problem.
However, I did notice one thing. I checked in 3 PDF files, and when I used Repository Manager to do a collection rebuild, I noticed that for Indexer Counters, the count for Full Text was 0 and the count for Meta Only was 3.
Anyone have any ideas? Is there something else that I missed? From reading the installation manual, it was not clear how database full-text indexing/searching would handle PDF files.

External URL Mapping and Full-Text Search Generation

I have a dilemma,
I can map external URL all fine e.g.
<map version="1.0">
<mapID target="Welcome" url="http://docweb.net/help/test.html"/>
</map>
but the html files are not contained in the local directory, and thus the jhindexer is unable to (as far as i know anyway) generate full-text search using the externally linked html files
also from using a decompiler I have found that you can "force" it to generate the help for external urls (as the indexer prefixes all urls with the "file:" protocol, but I can only get this to work under debug mode, else it gives a heapSpace exception, so I wish to avoid having to do anymore unneccesary digging/decompiling.
Also just as a gripe, why is most of the java help source code unavailable, and the Indexer seems to be very inflexible (ontop of not having any documentation)

That's the way it worked in X4 and later versions up but not including RH8. RH8 includes phrase searching in webhelp.
See www.grainge.org for RoboHelp and Authoring tips

TREX Full-Text search on Solution Directory

Hi Experts,
I have a problem with Full-Text search on the Solution Directory. Basically it doesn't work at all.
In transaction SOLUTION_MANAGER I choose a solution and then ->Operation Setup -> Solution Settings. At that point from the top menu I choose Solution Directory -> Find Document...
When I trigger the search using normal attributes the process finds documents related to this Solution without any problem. However, when I search using the "Full-Text Search" feature, the process doesn't find any document at all even though they exist.
Why is this functionality not working? Is this problem related to TREX?
What should i do to fix it? How do I create an index for these documents?
I haven't been able to solve this issue as TREX seams to work fine... can anybody help me out?
Any idea?
Many thanks,
Aldo

David:
I want to know if Oracle Context Option to need run over Oracle Application Server for to work search over multiple tables???
Exist some method for this (Context) run without to use Oracle Application Server???
select * from table_name where
contains(column_name1, 'keywords')>0 AND
contains(column_name2, 'keywords')>0However, using the Concatenated Datastore to create one index that has all the columns to be searched as sections will get better performance, since your query then only needs to use one contains statement across one index.
select * from table_name where
contains(indexed_column_name,
'(keywords WITHIN section_name1) AND
(keywords WITHIN section_name2) AND
keyords')>0Note: The last keyword in this query is so you search the indexed column as well, which is not a section. You could also create a dummy column to create the index on, and make all columns to search sections of that.
Cheers
David<HR></BLOCKQUOTE>
null

Kazehakase with full-text search in history using Hyper Estraier

A guide for Kazehakase with full-text history search using Hyper Estraier
I adopted qdbm and submitted hyperestraier in AUR, so you can enable full-text search frature by installing Hyper Estraier from AUR and rebuilding Kazehakase using srcpac or yaourt.
1. Install QDBM and Hyper Estraier from AUR. The easiest way is using yaourt. (If you prefer not to use yaourt, download tarball and do makepkg && pacman -U manually.)
yaourt -S qdbm hyperestraier
2. Rebuild Kazehakase using srcpac.
srcpac -Sb kazehakase
Of cource you can rebuild Kazehakase using yaourt.
yaourt -Sb kazehakase
You don't have to modify configure option in PKGBUILD of Kazehakase, because "--enable-hyper-estraier" is implied by default. If Hyper Estraier is installed successfully, you'll get "Hyper Estraier: yes" in configure messages.
3. Configure Kazehakase. To enable full-text search in history, run Kazehakase and go Edit>Preference>General and change UI Level to "Expert" and apply settings. Next, go Edit>Preference>History and set Search engine name to "hyper-estraier" and restart Kazehakase. Then you'll see "History Search" box next to "Internet Search" box.
Sorry for my poor English.

"ctxsrv" is no longer supported at version 10.1.
Instead PARAMETERS clause has SYNC option, and you can specify ON COMMIT for this.
If you created the database with DBCA and chose Oracle Text option, then you have no need to perform any further operations to configure Oracle Text.

Full text search for web ? Yes or no ?

Hi,
I have a DB that has more then 1.8mil records in a single table .... and would like to implement full text search or some sort of caching for quicker Web search ....
Let me describe you what I have .... The table that holds 1.8mil records is made out of 30 clob columns ... each holding text .... actually these are alphabetic columns ... so words that start with char 'A' are in the first clob ... 'B' in second 'C' in third and so forth ....
Searching is always done first by customerID and CreateDate which are both indexed columns , and then clobs are searched using instr ...
Execution plan was good ... but searching times started to increase ....
So therefor I would like to improve the search ... by implementing some sort of caching mechanism ....
I read a lot about this and found an example where I would create a table containing unique words and table for occurrences of the words ... but this would then
make like 1.8mil articles containing approc 500 words , which would then repeat through articles ... so ok there would be less then 50.000 unique words (in our language ) , but the occurrences would dramatically increase cause every word inside article has to have a link in occurrences table ... so this would then be like 900mil records inside table ..
Is this at all possible to have so many records inside single table ? And still make it quick ?
Is the Oracle Full text search the only right way in this situation ?
Any suggestions ? Did anyone implement anything like this ?
Thanks,
Kris

Let's start with your Oracle version. Please specify which version you run because Text capabilities vary dramatically between releases.
>
I tried using Oracle Text as suggested ... now if I understand correctly ....
CTXCAT - would be great because when new records are added, index is updated automatically .... but doesn't support CLOBs ... so no go
>
CTXCAT is a concatenated transactional index that is supposed to optimize combined searches on text and other columns. No go for you as it indeed does not support CLOB columns.
>
CONTEXT - supports CLObs , but I need to explicitly synchronize index ....
There are like 4000 inserts per day ..... and they all need to be indexed in a real-time ...
>
Not true, at least since 10g: SYNC(ON COMMIT) parameter makes this index type transactional (it's synchronized automatically on commit with this parameter set.)
>
If CTX_DLL.SYNC_INDEX procedure synchronize the whole table which is now 1.8mil records, this can take a while ... so it can't be run after inserts ....
>
It does not, it only synchronizes changed data since last sync operation.
So CONTEXT is actually perfectly suited for your needs (just redesign those 30 columns into one document column and index it.) Note that you need to regularly maintain CONTEXT indexes by scheduling CTX_DDL.OPTIMIZE_INDEX to run at off-hours and purge stale/removed data and rebuild its own internal index bitmaps for better performance. Otherwise you will see performance degrade as changes to the indexed data accumulate. You might also want to tweak initial indexing parameters, especially MEMORY parameter, as it greatly affects resulting index fragmentation - the more memory you give for initial indexing or optimization, the less fragmented and the more performant the index will be all other things equal.

Full Text searching number is slow

Hi
I configure the Full text search
some words like ' c\s full , m\s xxxxx like this because of \ it is not searching... if i replace to '' it is not searching the word that i require.
if i search the words that have number 432323XXX these words search are very slow
Please help
Thanks & Regards Manoj

Hi Manoj,
The performance of full-text queries could be influenced by various factors, such as memory, disk speed, fragmented catalog and so on. To improve the performance of full-text queries, please consider the following things.
• Reduce the number of fragments, reorganize the full-text catalog by using the REORGANIZE option of the ALTER FULLTEXT CATALOG Transact-SQL statement.
• Limit results and increase performance by using the top_n_by_rank parameter of the FREETEXTTABLE and CONTAINSTABLE functions.
• Check the full-text query plan to make sure that the appropriate join plan is chosen.
For more details, please review the following link.
http://msdn.microsoft.com/en-us/library/cc879244.aspx
Thanks,
Lydia Zhang

TREX/Content Server: Full text searches not working

I've setup TREX and Content Server in a sandbox environment and have been trying to get my head around DMS. I've completed what I've believed are the required steps (outlined below) and I have submitted 12 documents of various types into the respository. I'm able to perform meta data searches but not full text searched. Any ideas what I might have missed?
Much thanks!!!!
/Greg
0. Defined connection to TREX Server: TREX_42 (SM59); Using TREXADMIN all services are green.
1. "Define Document Area" - DMS is present; assumed I would use this one.
2. "Defined Indexing Readiness" - selected "Index Release" & "Classify" for DMS; set "Search Srv" to DMS_TEST
3. "Define Content Respository" - Created Z_CR_001 and maintained it as follows:
- Documentation Area: Document Management System
- Storage Type: HTTP content server
- HTTP Server: srv-sptxd2-dal.intervoice.int (content server)
- Port: 1090
- HTTP Script: ContentServer/ContentServer.dll
- Physical Path: /usr/sap/IRD/SYS/global/
4. "Define Content Categories" - Created ZCSSTRUCT; maintained as follows:
- Document Area: DMS
- Content Rep: Z_CR_001
5. Checked CSADMIN for Z_CR_001 - All Green
6. Went into SKPR07 (monitoring tab); entered DMS_PCD1 and "Indexing possible", "Auto indexing" and "DocArea activated" is all green.
7. Went into SMRO
- created SSR: "Search Server ID" = DMS_TEST, "Search engine" = DRFUZZY
- Verified "RFC Destinations"; passed tests; all green.
8. Went into SKPR07; using DMS_PCD1 as document class.
- executed "Clean up table entries"
- deselected "Limited to selected language"
- executed reindex. "Copied successfully from search engine"
- Selected the created index category
- Set the "Document area" indicator
- execute "Trigger process"; 12 documents referenced
9. Going into TREXADMIN I can see the created index; however the document count = 0. Also there is no queue created -- should there be?
Edited by: Greg Preston on Aug 5, 2009 9:11 PM

Dear
this link will help you,
TREX Architecture
http://help.sap.com/saphelp_erp60_sp/helpdata/EN/71/b26b2bfdc4eb47ab5432e8444290ce/content.htm
File Formats Supported by TREX
http://help.sap.com/saphelp_erp60_sp/helpdata/EN/55/cb634114b78047e10000000a1550b0/content.htm
Detail about TREX Configuration (Basic to advance)
http://help.sap.com/saphelp_erp60_sp/helpdata/EN/46/bab1d48b0a1514e10000000a114a6b/content.htm
Regards
Tushar Dave
Edited by: Tushar Dave on Aug 6, 2009 9:59 AM

Cannot do a full text search on pdf or Microsoft office Documents.

I am using Oracle Content Server 10gR3
Apart from the standard components and other components, I have InBound refinery Support and Oracle text Search Components enabled.
I get the following error when I check in a pdf or a Microsoft office document.
Text Conversion of the file '<repo_home>/.pdf' failed.*
Content has been indexed with Info only. Resubmit should only be performed if the problem has been resolved.
I am able to perform a full text search on *.txt and *.htm files.

Here are some more errors from Console output log
requestaudit     10.28 12:20:00.011     Audit Request Monitor     Request Audit Report over the last 120 Seconds****
requestaudit     10.28 12:20:00.011     Audit Request Monitor     -Num Requests 349 Errors 345 Reqs/sec. 2.911 Avg. Latency (secs)0.309 Max Thread Count 2
requestaudit     10.28 12:20:00.011     Audit Request Monitor     1     Service DOC_INFO     Total Elapsed Time (secs) 106.944     Num requests 345     Num errors 345     Avg. Latency (secs) 0.31
requestaudit     10.28 12:20:00.011     Audit Request Monitor     2     Service COLLECTION_GET_REFERENCE     Total Elapsed Time (secs) 0.701     Num requests 2     Num errors 0     Avg. Latency (secs) 0.35
requestaudit     10.28 12:20:00.011     Audit Request Monitor     3     Service COLLECTION_DISPLAY     Total Elapsed Time (secs) 0.211     Num requests 1     Num errors 0     Avg. Latency (secs) 0.211
requestaudit     10.28 12:20:00.011     Audit Request Monitor     4     Service COLLECTION_GET_INFO     Total Elapsed Time (secs) 0.02     Num requests 1     Num errors 0     Avg. Latency (secs) 0.02
requestaudit     10.28 12:20:00.011     Audit Request Monitor     ****End Audit Report*****

When can oracle support full text search for Simplified Chinese?

When I create index using the create index clause,the following errors appears
"ORA-29855: error occurred in the execution of ODCIINDEXCREATE routine
ORA-20000: interMedia Text error:
DRG-11440: operation not supported for the SIMPLIFIED CHINESE language"
Maybe I have to use like to query words...
Did somebody encounter the same problem and have a good solutions?I am expecting your help!
null

Hi,
Full-text search capabilities is not currently not supported and Unfortunately, we do not have a timeline as to when it will be available.
You can post a feedback on the below link.
http://feedback.azure.com/forums/217321-sql-database
Regards,
Mekh.

Disable Full Text Search in Records Management

Hi Everyone,
This is Shiva, we implemented SAP Records management & i am came in middle of this project. Right now client want to do some customizations in Records.
Actually i need to disable Full Text Search under Records. Under Cross Query i disabled the Full text by the following process
(Two sample element types are delivered with the service provider
SRM_SP_QUERY:
SRM_SPS_QUERY
Search for records, documents, cases, record models and file plans
SRM_SPS_QUERY_REC_DOC)
Under(SPRO) Cross SP search based on Property Unification; you can able to disable the FullText search field(this will be effects only for Cross Query), but under Records, when you right click there will be one option called FIND in which i am unable to disable the FULL TEXT search.
I used DMWB T-code under LOIO, i search for Standard/free/instance attributes, but i didn't find any Full Text Search Field in that one.
Can anyone please help me in this. I really appreciate your inputs & help.
Thanks,
Shiva.
Edited by: Shivasap on Aug 2, 2010 6:01 PM
Edited by: Shivasap on Aug 3, 2010 4:01 PM

Dear Shiva,
Full text search needs to be deactivated at content model level for attributes. You will find a document class in the element type of the record. In transaction DMWB, you will find LOIO & PHIO classes for the document class. For each attribute, you will find a checkbox "index-relevant" which needs to be unticked if you dont want to use TREX. For deactivating full-text search for which a field is given in the FIND popup, TREX needs to be deactivated. There is no option to deactivate just for a record element type in RM. Unfortunately, the field itself cannot be hidden. This is programmed in the logic.
Best Regards,
Pragya

Full text search in CMSK 9.0.3 WebStarterApp

I deployed WebStarterApp on CMSDK 9.0.3 and I cannot perform full text search on the CMSDK document.
Is there anybody who encounters the same problem ?
Thanks

I don't know how it was implemented in 9.0.2 or earlier. Sheduling the ctx_ddl.sync every minute will suffice for most projects. If you really want 'near real time' index synchronization, I guess you could implement the synchronization in S_TieContentObject (the java way) or perhaps create a trigger on odmz_contextrouter (the plsql-way). Keep in mind, however, that Oracle Text synchronization will allways be asynchronuous (and isn't part of the transaction), so you can't rely on Text to find content immediately after you've inserted it.

Full-text search containstable AND logical operator performance problem

Similar Messages

Maybe you are looking for