Implementing Multilingual Full Text Search

Hi All,
I am working on an application which requires to implement Full Text Search on Documents Uploaded in Binary Format. When I tested it first time with English Language, the search worked. But now the application demands to implement multilingual search on
that column. The document can be in Norwegian, Swedish, Danish or English language and then converted into binary.
When we create index for Full Text, we have to specify one language for that index. We also can't multiple indexes nor we can create multi-language indexes in same column.
Please help me out on how this requirement can be possibly met.
Thanks in advance.
Thanks and Regards, ErKakkar

Hello,
Full Text Search with SQL Server CE = Compact, the topic of this Forum? Or do you mean the "real" SQL Server?
You can use the neutral language, see
Choose a Language When Creating a Full-Text Index
Olaf Helper
[ Blog] [ Xing] [ MVP]

Similar Messages

Full Text Search in BDS

Hi all,
does anybody have a clue about implementing a full - text - search on Documents which are stored in the BDS.
The Goal is to offer a BSP Application with such kind of functionality. Documents are stored as Masterdata Documents in the BDS.
Are there any Classes or BAPIS for such kind of search ?
Many thanks in advance.
Regards,
Ali

i dont think you will find such a tool. but if you have maintained key words for your BDS documents, you can do a search on that. check out OAOR transaction to see how its implemented.
particularly the following program
submit bdsfind with classn = classn
 with classtyp = classtyp
 with objkey = objkey
 with descrip = descrip
 with contrep = contrep
 with docclass = docclass
 with doctype = doctype
 with creauser = creauser
 with chanuser = chanuser
 with key_1 = key_1
 with key_2 = key_2
 with key_3 = key_3
 with key_4 = key_4
 with key_5 = key_5
 with pfstatus = pfstatus
 and return.
Regards
Raja

Document Management / Full Text Search in Projects & SolMan Directory

Hello experts!
I want to implement Full Text Search in my SolMan and therefore need your help.
I have installed Trex 7.10 and am already able to search within my Solution Database (Problems/Solutions).
Now I want to be able to search within all documents in my SolMan Installation - e.g. Roadmaps, SAP Scenario Descriptions or my own uploaded documents.
In the IMG I already found the section Document Management but I cannot get the needed RFCs (IMSDEFAULT and IMSDEFAULT_REG) working.
There is already a RFC called IMSDEFAULT, which was created during the installation of Trex and runs the
Registered Server-Program: Trex_SOT_20080327090311
I tried to configure my RFCs according to the IMG but, as I said, cannot manage to get them working.
I hope anybody has an idea or an guide on how to implement Full Text Search!
Please be so kind and provide help - thanks in advance!
Best regards
Gerold

Gerold,
I am running into the same issues because we want to do full text search on project documentation within Solution Manager. The default RFC connections IMSDEFAULT_REG and IMSDEFAULT do not work with the standard settings.
We created one index for IWBSOLAR so maybe I also need to check IWBPROJ in SKPR06 for project documentation and create a new index for this one ?
The indexing for IWBSOLAR now fails although there seem to be over 6000 documents marked but none are put in the index. After indexing there is a list of the documents all with code 6401 but cannot find any where what this code means. It seems to be succesful as all entries are green.
Any help/suggestions on this topice would be greatly appreciated.

QUESTION OF FULL TEXT SEARCH IN R/3 4.6C USING TREX IN EP 6.0 SP9

Hi all:
I can use R/3(46c) plus content server(6.3) to store
Documents by KPRO.I've installed EP(6.0 sp9) plus Trex(6.1sp9).I want use this Trex to implement full text search,I can see the index id in the trex_xx\index directory.when I use tcode skpr07,I can find no problem,But when I use tcode cv04n,I can not find any document,the system does not display any error.
What's wrong?

hi there
can anyone help me with this...
How to integrate TREX in SAP R/3 System.
I found that u r discussing abt TREX in SAP R/3. Can u people help me with the basics required for that....
I have tried TREX in SAP Enterprise Portal and its working.
Now i would like to incorporate TREx in R/3 system.
How do we do that... Is it necessary that we need to install SAP Gateway in TREX host. Or is it possible without that.
Can anyone brief me on what are the important settings to be made for that.
Also i came across the RFC destination creation for connecting TREX with the R/3 system.
I would like to know what kind of data is commonly indexed in R/3 and that too how. What is it that provides an interface for the user.
In the case of portal we define a a KM repository as the datasource likewise in R/3 system what do we define as datasource.
I know these questions may sound very silly... pls do help
regards
Sasha

Information on full text search in Oralce Database

Hi,
We are looking to implement full text search using Oracle database. Where can I find info on this topic? Specifically, I'm looking for
1) an overview of how to implement them in Oracle database - column type, size/limitations etc.
2) does oracle database come with filters to extract and filter data from different file formats such as Ms office, PDF (images) etc.
Appreciate your reply

Look into the Oracle Text documentation. It has the answers to your questions.

Full text search in Filenet via SAP DMS

Hello experts,
Hopefully you can help me with the following. My customer has SAP DMS and uses FileNet (v4.0) as content repository. They use Verity as tool for full text search (SAP and non-SAP). Verity has it's own user interface / search screen. Question is if it is possible for SAP (easy)DMS users to do full text search (via CV04N or EasyDMS searchscreen) and search in the FileNet repository (directly with TREX or via Verity). The customer has the ACSAP connector up and running.
I've implemented TREX with SAP content server and index server before, but I'm not familiar with FileNet possibilities in that area, so hopefully someone has experience with this?
thanks in advance, Erwin

Dear NAG
For text search you have activate TREX Server
follow the steps
Set TREX
A:- Transaction: SRMO
Install TREX , and create RFC destination to connect to
TREX.
B:- Transaction: SKPR06
Check the flag to Document Area "DMS" to utilize this document area for
document search.
For Indexing
A: Transaction: SE38
Run the report program "RSTIRIDX_REINDEX" to create Index.
Also, run the report program "RSTIRIDX" to create Index.
Set mime type for the text search
A: SPRO --
Cross-Application Components > Document Management System > General
Data > Settings for Storage Systems > Create MIME types for full
text search,
Enter MIME type like
"application/msword",
"application/pdf".
Retrive document via Test Search
A: Then, test retrieval document search using
Transaction: SKPR07.
Enter Document Class "DMS_PCD1", Language "EN", Document Class "DMS",
and try "TEST search".
Regards
Tushar Dave

Using Punctuations in Full Text Searches

Hi, I'm having a bit of trouble implementing full text search effectively in regards to punctuations.
In my thesaurus file I have something along the lines of
<expansion>
intl
int'l
international
</expansion>
and when I run the query
select * from sys.dm_fts_parser('Formsof(freetext,"international")',1033,null,0)
it show's that it is included
international's
internationals
internationals'
international
intl
int'l
However when I do end up running the query using Contains, it ignores everything that contains int'l and only returns the results of the other matches.

I am moving it to Search.
Kalman Toth Database & OLAP Architect
SQL Server 2014 Design & Programming
New Book / Kindle: Exam 70-461 Bootcamp: Querying Microsoft SQL Server 2012

[iPhone SDK] Full text search in SQLite

I'm trying to implement a search feature on my SQLite-stored data. Basically, the user gives me a list of keywords; I'd like to retrieve records which have all the keywords in at least one of two fields, and which also have a foreign key set to a specific value. As far as I can tell, the iPhone doesn't include SQLite's full-text search extension. Does anyone have a suggestion as to how I should implement this feature? My current idea is to simply create a table that maps every word in the record back to the record itself, but that might be a wasteful approach...

I got SQLite working by getting the SQLite source
(http://sqlite.org/sqlite-amalgamation-360.zip)
added
#define SQLITEENABLEFTS3 1
at the top of sqlite3.c
and added both sqlite3.c and sqlite3.h into the project.
Make sure the standard SQLite framework has been removed.
Build & Go and everything working perfectly with the latest SQLite and Full Text Search built in.
(No idea if compiling and linking is preferential)

Disable Full Text Search in Records Management

Hi Everyone,
This is Shiva, we implemented SAP Records management & i am came in middle of this project. Right now client want to do some customizations in Records.
Actually i need to disable Full Text Search under Records. Under Cross Query i disabled the Full text by the following process
(Two sample element types are delivered with the service provider
SRM_SP_QUERY:
SRM_SPS_QUERY
Search for records, documents, cases, record models and file plans
SRM_SPS_QUERY_REC_DOC)
Under(SPRO) Cross SP search based on Property Unification; you can able to disable the FullText search field(this will be effects only for Cross Query), but under Records, when you right click there will be one option called FIND in which i am unable to disable the FULL TEXT search.
I used DMWB T-code under LOIO, i search for Standard/free/instance attributes, but i didn't find any Full Text Search Field in that one.
Can anyone please help me in this. I really appreciate your inputs & help.
Thanks,
Shiva.
Edited by: Shivasap on Aug 2, 2010 6:01 PM
Edited by: Shivasap on Aug 3, 2010 4:01 PM

Dear Shiva,
Full text search needs to be deactivated at content model level for attributes. You will find a document class in the element type of the record. In transaction DMWB, you will find LOIO & PHIO classes for the document class. For each attribute, you will find a checkbox "index-relevant" which needs to be unticked if you dont want to use TREX. For deactivating full-text search for which a field is given in the FIND popup, TREX needs to be deactivated. There is no option to deactivate just for a record element type in RM. Unfortunately, the field itself cannot be hidden. This is programmed in the logic.
Best Regards,
Pragya

Full-text search in ucm10g with wc search taskflow

hi all,
I am using wc11.1.13 plus ucm10gr3.
I checked in a content with both Primary File and Alternate File , I used webcenter search taskflow to search word in alternate file, but search result show me the Primary File, how to resolve this issue?
I also opened a case in ucm forum.
Best regards

Hi Atrsams,
According to your description, you want to implement Full Text Search in Analysis Services Cube, right? Base on my research, there is no such a functionally to achieve Full Text Search in Analysis Services currently. And it seems that there is no any
solution or workaround for it. I recommend you to submit this at
https://connect.microsoft.com/SQLServer/ So that the product team can consider to solve it in the next SQL Server version. Your feedback is valuable for us to improve our products and increase the level
of service provided.
Thanks for your understanding.
Regards,
Charlie Liao
TechNet Community Support

Full text search in CMSK 9.0.3 WebStarterApp

I deployed WebStarterApp on CMSDK 9.0.3 and I cannot perform full text search on the CMSDK document.
Is there anybody who encounters the same problem ?
Thanks

I don't know how it was implemented in 9.0.2 or earlier. Sheduling the ctx_ddl.sync every minute will suffice for most projects. If you really want 'near real time' index synchronization, I guess you could implement the synchronization in S_TieContentObject (the java way) or perhaps create a trigger on odmz_contextrouter (the plsql-way). Keep in mind, however, that Oracle Text synchronization will allways be asynchronuous (and isn't part of the transaction), so you can't rely on Text to find content immediately after you've inserted it.

Full text search in SSAS

Hi All,
If anyone has implemented full text search in analysis services, please help me on this. I know we can implement this in SQL Server, but I am not sure on analysis services. If full text is not possible to implement in analysis services, at least please let
me know a work around.
Thanks,
ATRSAMS

Hi Atrsams,
According to your description, you want to implement Full Text Search in Analysis Services Cube, right? Base on my research, there is no such a functionally to achieve Full Text Search in Analysis Services currently. And it seems that there is no any
solution or workaround for it. I recommend you to submit this at
https://connect.microsoft.com/SQLServer/ So that the product team can consider to solve it in the next SQL Server version. Your feedback is valuable for us to improve our products and increase the level
of service provided.
Thanks for your understanding.
Regards,
Charlie Liao
TechNet Community Support

Is it possible to ignore noise words conditionally in working with Full text search containstable

I have a question on stoplist file. I need to search for exact phrase string("this is the incident") which contains noise words. As part of the FT search engine, during parsing it eliminates noise word and search on remaining string in
the given phrase.
let us say there are 10 rows which contains the term "incident" in the FT table . and 1 row which has the exact phrase.i.e."this is the incident".
if we use containstable() to search for "this is the incident", we are getting 10 rows instead of 1 row.
To resolve the issue, we have 3 solutions
1.either stoplist file needs to be modified to remove the words (this,is,the)
2. set stoplist = OFF.
3.empty stoplist.
Apart from the above solutions, is there any better solution with out touching noise words file list.
If any solution that provides flexibility to ignore noise words conditionally at one time and not to ignore them.
Please provide your suggestion.
kkprasad

One question that I ask is: Why would I want to exclude noise words?
Noise words were created to limit the size of the full text indexes and avoid processing the many 'this', 'is', and 'the' common words. But the disadvantage of doing so is that you cannot find some things as you would like.
My feeling is that computers are more powerful and have more storage and it is often better to just index everything. As long as your search does not include 'the', then the large number of 'the's in the system will pretty much be ignored.
NOTE: If you change the noise words, including SET STOPLIST = OFF, you have to rebuild the index in order for it to implement your decision.
Of course, for very, very large full text indexes you would need to test.
Is your full text search on relation database columns, e.g. Description NVARCHAR(1000) or are you searching Word, Excel, and other more complex data?
If your full text is relational columns, it might be that you could:
1. Select only the fulltextkey into a temp table (e.g. #FTSfulltextkey) from the full text index using noise words. That would give you 10 rows.
2. Then directly query the table to find the string as you define above. (But remember that punctuation and symbols are generally ignored by Full Text Indexing, but would still be there in the string of text.)
SELECT *
FROM MyTextTable T
JOIN #FTSfulltextkey K
ON T.fulltextkey = k.fulltextkey
WHERE T.Description like '%this is the incident%'
Full text search is powerful, but it has limits. And the behaviour changes depending on the Language of the search.
RLF

Full text search for web ? Yes or no ?

Hi,
I have a DB that has more then 1.8mil records in a single table .... and would like to implement full text search or some sort of caching for quicker Web search ....
Let me describe you what I have .... The table that holds 1.8mil records is made out of 30 clob columns ... each holding text .... actually these are alphabetic columns ... so words that start with char 'A' are in the first clob ... 'B' in second 'C' in third and so forth ....
Searching is always done first by customerID and CreateDate which are both indexed columns , and then clobs are searched using instr ...
Execution plan was good ... but searching times started to increase ....
So therefor I would like to improve the search ... by implementing some sort of caching mechanism ....
I read a lot about this and found an example where I would create a table containing unique words and table for occurrences of the words ... but this would then
make like 1.8mil articles containing approc 500 words , which would then repeat through articles ... so ok there would be less then 50.000 unique words (in our language ) , but the occurrences would dramatically increase cause every word inside article has to have a link in occurrences table ... so this would then be like 900mil records inside table ..
Is this at all possible to have so many records inside single table ? And still make it quick ?
Is the Oracle Full text search the only right way in this situation ?
Any suggestions ? Did anyone implement anything like this ?
Thanks,
Kris

Let's start with your Oracle version. Please specify which version you run because Text capabilities vary dramatically between releases.
>
I tried using Oracle Text as suggested ... now if I understand correctly ....
CTXCAT - would be great because when new records are added, index is updated automatically .... but doesn't support CLOBs ... so no go
>
CTXCAT is a concatenated transactional index that is supposed to optimize combined searches on text and other columns. No go for you as it indeed does not support CLOB columns.
>
CONTEXT - supports CLObs , but I need to explicitly synchronize index ....
There are like 4000 inserts per day ..... and they all need to be indexed in a real-time ...
>
Not true, at least since 10g: SYNC(ON COMMIT) parameter makes this index type transactional (it's synchronized automatically on commit with this parameter set.)
>
If CTX_DLL.SYNC_INDEX procedure synchronize the whole table which is now 1.8mil records, this can take a while ... so it can't be run after inserts ....
>
It does not, it only synchronizes changed data since last sync operation.
So CONTEXT is actually perfectly suited for your needs (just redesign those 30 columns into one document column and index it.) Note that you need to regularly maintain CONTEXT indexes by scheduling CTX_DDL.OPTIMIZE_INDEX to run at off-hours and purge stale/removed data and rebuild its own internal index bitmaps for better performance. Otherwise you will see performance degrade as changes to the indexed data accumulate. You might also want to tweak initial indexing parameters, especially MEMORY parameter, as it greatly affects resulting index fragmentation - the more memory you give for initial indexing or optimization, the less fragmented and the more performant the index will be all other things equal.

Full-Text search is not working with PDF files - SQL Server 2012 64 bit

Hi,
We are in the process of storing PDF files in SQL Server 2012 with Full-Text search capability.
I followed the steps as below and it works fine with word document but not for PDF files. I tried with PDF ifiler 11 & 9 and both are unsuccessful.
Server/DB Level Settings:
1)
Enable FileStream
2)
Install Full-Text
then restart
3)
Use [specific db]
alter
database [db name]
add
filegroup Files
contains filestream;
alter
database [db name]
add
file (
name = N'Files',
filename =
N'D:\SQL\DATA') to
filegroup [Files];
3)
Database level
Settings:
FileStream:
FileStream
Directory name:
[Set the name]
FileStream
non-transacted
Access: [set Appropriate]
3a)
Add a
datafile to DB
with filestreamdata
filetype.
4)
Share D:\SQL\DATA
directory and
add specific accounts
with read/write
access
5)
Give bulkadmin
access to those
specific accounts
at server
level
6)
From the
page (link)
download and
install the *.pdf
IFilter for
FTS. Link:
http://www.adobe.com/support/downloads/detail.jsp?ftpID=5542
7)
To the
PATH global system
variable add
path to the
catalog,
where you installed
the plugin.
Default for
this version is:
C:\Program
Files\Adobe\Adobe
PDF iFilter 9
for 64-bit
platforms\bin
8)
From the
page (link)
download a
FilterPackx64.exe
and install
it. Link:
http://www.microsoft.com/en-us/download/confirmation.aspx?id=20109
9)
Now from
SSMS execute the following
procedures:
-sp_fulltext_service
'load_os_resources',1
-sp_fulltext_service
'verify_signature', 0
EXEC
sp_fulltext_service
'update_languages';
-- update language list
EXEC
sp_fulltext_service
'restart_all_fdhosts';
-- restart daemon
reconfigure
with override;
10)
Restart the
server
11)
select document_type,
path from
sys.fulltext_document_types
where document_type
= '.pdf'
-select
document_type,
path from sys.fulltext_document_types
where document_type
= '.docx'
12) Results are OK.
Following is my Table /Index/ catalog script:
CREATE
TABLE dbo.DocumentFilesTest
DocumentId INT
IDENTITY(1,1)
NOT NULL
PRIMARY KEY,
AddDate datetime
NOT NULL,
Name nvarchar(50)
NOT NULL,
Extension nvarchar(10)
NOT NULL,
Description nvarchar(1000)
NULL,
FileStream_Id UNIQUEIDENTIFIER
ROWGUIDCOL NOT
NULL UNIQUE DEFAULT
NEWSEQUENTIALID(),
FileSource varbinary(MAX)
FILESTREAM DEFAULT(0x)
go
--Add default add date for document
ALTER
TABLE dbo.DocumentFilesTest
ADD CONSTRAINT
DF_DocumentFilesTest_AddDate
DEFAULT sysdatetime()
FOR AddDate
EXEC
sp_fulltext_database
'enable'
GO
IF
NOT EXISTS
(SELECT
TOP 1 1 FROM sys.fulltext_catalogs
WHERE name
= 'Ducuments_Catalog_test')
BEGIN
EXEC sp_fulltext_catalog
'Ducuments_Catalog_test',
'create',
'D:\SQL\PDFBlob';
END
--EXEC sp_fulltext_catalog 'Ducuments_Catalog_test', 'drop'
DECLARE
@indexName nvarchar(255)
= (SELECT
Top 1 i.Name
from sys.indexes
i
Join sys.tables
t on
i.object_id
= t.object_id
WHERE t.Name
= 'DocumentFilesTest'
AND i.type_desc
= 'CLUSTERED')
PRINT @indexName
EXEC
sp_fulltext_table
'DocumentFilesTest',
'create',
'Ducuments_Catalog_test',
@indexName
EXEC
sp_fulltext_column
'DocumentFilesTest',
'FileSource',
'add', 0,
'Extension'
EXEC
sp_fulltext_table
'DocumentFilesTest',
'activate'
EXEC
sp_fulltext_catalog
'Ducuments_Catalog_test',
'start_full'
ALTER
FULLTEXT INDEX
ON [dbo].[DocumentFilesTest]
ENABLE
ALTER
FULLTEXT INDEX
ON [dbo].[DocumentFilesTest]
SET CHANGE_TRACKING
= AUTO
ALTER
FULLTEXT CATALOG
Ducuments_Catalog_test REBUILD
WITH ACCENT_SENSITIVITY=OFF;
INSERT
INTO DocumentFilesTest(Extension,
Name,
FileSource)
SELECT
'pdf'
'BOL12006553.pdf'
* FROM
OPENROWSET(BULK
'd:\SQL\PDFBlob\BOL12006553.pdf',
SINGLE_BLOB)
AS BLOB;
GO
INSERT
INTO DocumentFilesTest(Extension,
Name,
FileSource)
SELECT
'docx'
'test.docx'
* FROM
OPENROWSET(BULK
'd:\SQL\PDFBlob\test.docx',
SINGLE_BLOB)
AS Document;
GO
SELECT
d.*
FROM dbo.DocumentFilesTest
d WHERE
Contains(d.FileSource,
'BILL')
Returns nothing. it should come from PDF file
SELECT
d.*
FROM dbo.DocumentFilesTest
d WHERE
Contains(d.FileSource,
'TEST')
Returns from word document as follows:
2           2014-06-04 10:11:41.393            test.docx docx
NULL   [BINARY Value] [Binary Value]
Any help is appreciated. Its been a long wait.
Thanks,
Vel
Vel Thavasi

Hello,
Did you check the fulltext log files for more details about the errors. If the filter isn’t working, there should be errors in the error log file.
The following thread is about similar issue, please refer to:
http://social.msdn.microsoft.com/forums/sqlserver/en-US/69535dbc-c7ef-402d-a347-d3d3e4860d72/sql-server-2008-64bit-fulltext-indexing-pdf-not-working-cant-find-ifilter
Regards,
Fanny Liu
If you have any feedback on our support, please click here.
Fanny Liu
TechNet Community Support

Implementing Multilingual Full Text Search

Similar Messages

Maybe you are looking for