Text & Keyword analysis

Is there anyway I can take a string or selection of text and somehow determine which words within the text are contextually significant? For example, I want to write a program in java that will take in text files and pick out the important keywords and ignore unimportant words (and, but, if for example).
Are there any means or methods developed to do this sort of thing, or any rules of thumb to conduct such analysis? I looked at pattern matching with regular expressions but its not quite what I am looking for (although it is the closest I have come yet), so can anyone advise on the best way to do this or how I may go about constructing such a program?

Yes, I agree. Deciding how "proficient" it should be is almost as tough as programming it! I think simply itterating through the docuements removing individual words from a pre-defined list is a bit too basic, but obviously full language understanding is an as yet unachievable AI task - is there even any middle ground? I'm looking for somewhere inbetween, leaning toward the basic side though! ;)
It doesn't need to put everything into context, merely pick out maybe just one or two important words in the docuement...

Similar Messages

Text/keyword searching

I have to search the comments field of a very large database (millions of records) for a large set of keywords. I have about 16,000 keywords that I am looking for. They are serial numbers, so I don't have to worry misspellings and spacing, whatever. The comments section will have other text as well. So if it was just one records, I could SELECT * FROM bigtable WHERE comment LIKE '%target%'. If I have two targets, I could use SELECT * FROM bigtable WHERE (comment LIKE '%target1%' OR comment LIKE '%target2%'). But I have 16,000 targets. I don't think my method scales very well (sarcasm).
Is there a way for me to do this?

Hi,
MrGibbage wrote:
Frank,
I actually have a question about your answer. In the LIKE statement, you have the word 'target'. What will that refer to in the query? target isn't defined anywhere, so I don't see how it will work. Can you please take one more look at this? Then I will buy you a virtual beer somewhere.Sorry, I should have explained that.
When I suggested you create a table of keywords, I meant something like this:
CREATE TABLE keywords
( target VARCHAR2 (20)
);You can add other columns if you have a need for them (e.g., if there are different categories of keywords, and sometimes you want to check only certain categories).
So target is the name of the column in the new table, called keywords, which will have 16,000 rows. Of course, there's nothing special about the names keywords or taget; use any valid names that make sense to you.

Change the Text of Analysis Web Item

Hi,
In one of my Querry, I craeted a formula. There it is checking some condition and according to that condition it display 1 or 0.
But in our project we need YES instead of 1 and NO instead of 0.
I can't find any solution to implement this in Query Designer.
So I tried to implement this through Web Application Designer. Means It will check that perticular field and change accordingly.
So if any have any idea regaring this then please Reply.
It's Urgent.

Hi Subranshu,
check the following link,
/people/andrey.uryukin/blog/2010/08/29/formatting-bex-queries-output-with-wad-script-webitem
In that they used inserted some text in the cell to get rid of # symbol,
You can apply that code to your scenario.
Regards,
Ranganath

How to update TEXT in Sales order header

Hello ,
I have requirement in which I am creating SO via IDOC calling BAPI_SALESORDER_CREATEFROMDAT2.
Question: Can any one help me in finding a solution to update the TEXT in sales order header which is confogured IN TEXT CONTROL in SD.
I have tried various option using CREATE_TEXT but it is only updating STXH table , the text is not coming in the sales order.
If someone knows about this , <removed by moderator>.
Edited by: Thomas Zloch on Nov 15, 2010 4:20 PM - please do assign priorities > normal

Sounds like a text determination config issue - have you assigned the text in VOTXN? Have you run the text determination analysis from VA02 in the document?

Customer Master - Sales text copy to Sales Document Header

Hello,
I have properly configured the text types, access sequences, and text procedures to take text from the Customer - Sales & Distribution and copy to the Sales Document Header. After testing this in my DEV system, everything is working appropriately.
I have transported these changes to my QA system, all transports went through successfully. All configuration is now the same between my DEV and QA systems, and the access sequences are recognized in both systems via the TTXERN table.
All changes were made using transaction VOTXN.
The problem is that when performing my test in QA, the text is not copied from the Customer Master - Sales Area text to the Sales Document Header text. When going to the Text Determination Analysis view, it states 111 - Access Sequence does not exist. When selecting Customizing, it clearly shows that the access sequences do exist.
Has anyone encountered this issue? I have checked SAP notes but have not found a proper resolution for this issue.
Edited by: Andy Slania on Jan 31, 2011 9:30 PM

Read OSS Note 1002455 - Errors in texts: Messages VX 201, VX 111, and VX 205 and Note 1002664 - FAQ: Text determination problems after upgrade.
Also check OSS Note 581779 - Text determination analysis: Access sequence does not exist.
One more OSS Note 970153 - Change to Customizing text not transferred to target system may also apply to your scenario.
Regards,

Is the word "text" an operator or reserved word in Oracle Text?

DB: Oracle 10.2.0.3.0
I have created a CONTEXT index on a table which uses a Multi-Column Datastore on two columns (text and keywords)
When I perform this query, I'm getting back every row in the table even though none contain a value of "text" within the TEXT or KEYWORDS columns.
SELECT *
FROM tb_rh_links
WHERE CONTAINS(search_column, 'text') > 0
** UPDATE ** it does the same things when searching for word "keywords." I'm beginning to think it has something to do with the section groups...
SELECT *
FROM tb_rh_links
WHERE CONTAINS(search_column, 'keywords') > 0
Here is how the index and multi-column datastore is setup:
BEGIN
ctx_ddl.create_preference ('rh_lex', 'basic_lexer');
ctx_ddl.set_attribute ('rh_lex', 'index_stems', 'english');
ctx_ddl.create_preference('rh_links_mcds', 'MULTI_COLUMN_DATASTORE');
ctx_ddl.set_attribute('rh_links_mcds', 'columns', 'text, keywords');
END;
BEGIN
ctx_ddl.create_section_group('rh_links_sg','basic_section_group');
ctx_ddl.add_field_section(group_name => 'rh_links_sg',
section_name => 'text',
tag => 'text');
ctx_ddl.add_field_section(group_name => 'rh_links_sg',
section_name => 'keywords',
tag => 'keywords');
END;
CREATE INDEX TB_RH_LINKS_IDX5 ON TB_RH_LINKS (KEYWORDS)
INDEXTYPE IS ctxsys.CONTEXT
PARAMETERS('datastore rh_links_mcds section group rh_links_sg');
Edited by: Dishoom on Apr 20, 2009 11:07 AM

When you create a multi_column_datastore without specifying a delimiter, by default the column names are tokenized and indexed. If you specify the newline delimiter, then the column names are not tokenized. Or, once you have created sections and used the section group in your parameters, then the column names are not tokenized. You may have created your multi_column_datastore, then created your index, tokenized the column names, then added the sections, but the tokenized column names weren't dropped until you recreated the index. Please see the demonstration below.
SCOTT@orcl_11g> CREATE TABLE tb_rh_links
2    (text            VARCHAR2(10),
3      keywords       VARCHAR2(10),
4      search_column VARCHAR2( 1))
5 /
Table created.
SCOTT@orcl_11g> INSERT ALL
2 INTO tb_rh_links VALUES ('word1', 'word2', null)
3 INTO tb_rh_links VALUES ('word3', 'word4', null)
4 SELECT * FROM DUAL
5 /
2 rows created.
SCOTT@orcl_11g> BEGIN
2    ctx_ddl.create_preference ('rh_lex', 'basic_lexer');
3    ctx_ddl.set_attribute ('rh_lex', 'index_stems', 'english');
4    ctx_ddl.create_preference('rh_links_mcds', 'MULTI_COLUMN_DATASTORE');
5    ctx_ddl.set_attribute('rh_links_mcds', 'columns', 'text, keywords');
6 END;
7 /
PL/SQL procedure successfully completed.
SCOTT@orcl_11g> CREATE INDEX TB_RH_LINKS_IDX5 ON TB_RH_LINKS (search_column)
2 INDEXTYPE IS ctxsys.CONTEXT
3 PARAMETERS('datastore rh_links_mcds')
4 /
Index created.
SCOTT@orcl_11g> -- the column names are tokenized:
SCOTT@orcl_11g> COLUMN token_text FORMAT A10
SCOTT@orcl_11g> SELECT token_text, token_first, token_last, token_count
2 FROM   dr$tb_rh_links_idx5$i
3 /
TOKEN_TEXT TOKEN_FIRST TOKEN_LAST TOKEN_COUNT
KEYWORDS             1          2           2
TEXT                 1          2           2
WORD1                1          1           1
WORD2                1          1           1
WORD3                2          2           1
WORD4                2          2           1
6 rows selected.
SCOTT@orcl_11g> SELECT *
2 FROM tb_rh_links
3 WHERE CONTAINS(search_column, 'text') > 0
4 /
TEXT       KEYWORDS   S
word1      word2
word3      word4
SCOTT@orcl_11g> SELECT *
2 FROM tb_rh_links
3 WHERE CONTAINS(search_column, 'keywords') > 0
4 /
TEXT       KEYWORDS   S
word1      word2
word3      word4
SCOTT@orcl_11g> -- if you use the newline delimieter, the column names are not tokenized:
SCOTT@orcl_11g> BEGIN
2    CTX_DDL.SET_ATTRIBUTE ('rh_links_mcds', 'DELIMITER', 'NEWLINE');
3 END;
4 /
PL/SQL procedure successfully completed.
SCOTT@orcl_11g> DROP INDEX tb_rh_links_idx5
2 /
Index dropped.
SCOTT@orcl_11g> CREATE INDEX TB_RH_LINKS_IDX5 ON TB_RH_LINKS (search_column)
2 INDEXTYPE IS ctxsys.CONTEXT
3 PARAMETERS('datastore rh_links_mcds')
4 /
Index created.
SCOTT@orcl_11g> SELECT token_text, token_first, token_last, token_count
2 FROM   dr$tb_rh_links_idx5$i
3 /
TOKEN_TEXT TOKEN_FIRST TOKEN_LAST TOKEN_COUNT
WORD1                1          1           1
WORD2                1          1           1
WORD3                2          2           1
WORD4                2          2           1
SCOTT@orcl_11g> SELECT *
2 FROM tb_rh_links
3 WHERE CONTAINS(search_column, 'text') > 0
4 /
no rows selected
SCOTT@orcl_11g> SELECT *
2 FROM tb_rh_links
3 WHERE CONTAINS(search_column, 'keywords') > 0
4 /
no rows selected
SCOTT@orcl_11g> -- or, with the section groups, the column names are not tokenized:
SCOTT@orcl_11g> BEGIN
2    -- return to default:
3    CTX_DDL.SET_ATTRIBUTE ('rh_links_mcds', 'DELIMITER', 'COLUMN_NAME_TAG');
4 END;
5 /
PL/SQL procedure successfully completed.
SCOTT@orcl_11g> BEGIN
2    ctx_ddl.create_section_group('rh_links_sg','basic_section_group');
3    ctx_ddl.add_field_section(group_name => 'rh_links_sg',
4       section_name => 'text',
5       tag => 'text');
6    ctx_ddl.add_field_section(group_name => 'rh_links_sg',
7       section_name => 'keywords',
8       tag => 'keywords');
9 END;
10 /
PL/SQL procedure successfully completed.
SCOTT@orcl_11g> DROP INDEX tb_rh_links_idx5
2 /
Index dropped.
SCOTT@orcl_11g> CREATE INDEX TB_RH_LINKS_IDX5 ON TB_RH_LINKS (search_column)
2 INDEXTYPE IS ctxsys.CONTEXT
3 PARAMETERS('datastore rh_links_mcds section group rh_links_sg')
4 /
Index created.
SCOTT@orcl_11g> SELECT token_text, token_first, token_last, token_count
2 FROM   dr$tb_rh_links_idx5$i
3 /
TOKEN_TEXT TOKEN_FIRST TOKEN_LAST TOKEN_COUNT
WORD1                1          1           1
WORD2                1          1           1
WORD3                2          2           1
WORD4                2          2           1
SCOTT@orcl_11g> SELECT *
2 FROM tb_rh_links
3 WHERE CONTAINS(search_column, 'text') > 0
4 /
no rows selected
SCOTT@orcl_11g> SELECT *
2 FROM tb_rh_links
3 WHERE CONTAINS(search_column, 'keywords') > 0
4 /
no rows selected
SCOTT@orcl_11g>

Material Sales Text.

I have run across something that I can not find an answer to. I have searched this Forum, OSS Notes, and tried to Google this as well. To make a long story short, we no longer have a SD Functional/Technical person and I am trying to learn as I go. When creating a Return (Doc Type RE), I enter a material which has no Sales Text for any of the Sales Orgs, but yet when I look at the Texts, I see a Material Sales Text. I have gone to the Material Master, added a Sales Text for the Sales Org I am working with, and the Text is bought in as I expect it to be on the RE. I then delete the Text, add the material to the RE, and again a Sales Text is being bought in even though no Sales Text exits in the Sales Org Views.
We are curerntly using version 4.7
Any help would be greatly appreciated. Thanks.

Go to the Sales Order and then on the line item (or the header if you see the text there) and choose the Texts tab.
Then choose the Text and clic on the "log" button on the bottom of the text panel.
This will show you the basic text determination analysis and will tell exactly from where this text came from

Web positioning - no text in iweb

I used iweb template, did not use their menus and organised my website that way. It is for my sister's restaurant and postioning is a real issue.
does anyone know how I can add text that would be searched by spiders, how I can create metatags and add keywords? When the site is completely finished, I will go in the HTML source and add all this by hand unless someone has a better idea. If it doesn't work, i will recreate the site with freeway, but it is a fair amount of work.
Has anyone tried to add simple text at the bottom of each page that would simply be a summary of the page in plaintext so that search engine would get some food? I know that hiding text playing with color is not a good idea but what about straight plain text?
any expert who could shed some light?
my web, nearly all translated in english, french and dutch, except the menus is at:
http://www.lahuttelurette.be
iBookg4-750MB, iMacG5-1MB Mac OS X (10.3.8)

Bernard,
Hello again! I'm glad to see that you've got all those flags working! Like I've told you before....very classy looking site. In terms of what you are asking, check this thread out...
http://discussions.apple.com/thread.jspa?threadID=525376&tstart=0
This fellow put plain text keywords on his pages and then made them "invisible" by matching the color of the text and background. I think this method would probably work in your favor. Keep in mind that this doesn't determine if or when your site will be found by the search engines, but it may provide simpler bits of information that may affect your ranking once the search engines have found you.
In terms of preventing your plain text from getting converted to an image, I guess changing the color would not trigger this judging by this most recent report. Try it out and let us know. Good luck to you, Bernard!
If you find this information useful, please take the time to provide me with feedback by marking my reply as "solved" or "helpful" using those little buttons that you see in the title bar of this reply. I'd really appreciate it!

Using multi-word keywords in "Find"

Let's say I have a keyword: "Downtown LA", and another keyword, "Downtown SLC". If I use the Lightroom "Find" option to search for "Downtown LA", it will actually find any images with "Downtown" as part of its keywording, including the "Downtown SLC" images.
I tried wrapping the keyword phrase in quotes to see if that would give me an exact phrase search, but it doesn't work that way apparently. I know I could drop down to the keyword browser, but let's say I'm looking for all images with both the "Downtown LA" keyword, AND the "Family Vacation" keyword. How would I search for such an intersection when the "find" menu locks in on individual words rather than the entire keyword phrase?

>You might want to check out the Keyboard Shortcuts and Search Rules document in the FAQ on this forum. There is a section which contains all the search arguments which allow you to do some pretty fancy stuff, including finding just "Salt Lake City"
I have a copy of that...it doesn't work.
Text: Keywords
Rule: Contains All
Text Field: Black Canyon
What I expect are photos with a keyword containing "black" and "canyon".
What I get are any photo with a keyword or any of its parents containing "black" and a keyword or any of its parents containing "canyon".
So, in addition to photos of Black Canyon Dam and photos taken in Back Canyon I also get photos of Black Birds taken in Canyon County.
This can be "fixed" by changing "Black birds" to "Blackbirds" and surrounding "Black" in the text field with plus signs...this is silly.
The space character is a ligitimate keyword character and ought to be treat as such if it does not appear at the beginning or end of a keyword.
The keyword entry panel uses commas as delimiters. Why are spaces used as delimiters in the Find panel?
How about the same delimiter (a comma) everywhere?
Winston Mitchell
Boise, Idaho

Keyword search in 3.3rc not working.

Has anyone else been unable to make a successful keyword search after updating to 3.3 rc?
Solution?
Thanks,
BE

Thanks for your reply. I'm Mac. Searching the only way I know how, backslash to bring up search options at the top of Library desktop, select text > keywords > contain all, then enter the keyword. Only a few pics show up whether I use 'keywords' or 'any searchable field.' I too thought there would be more chat about it if it was a major problem. Must be something weird with my set up. Needing to work against a deadline, I returned to 3.2 where it works perfectly.
Again, thank for your reply.

[tip] search for / selecting NO keywords (selection photos without a keyword)

Probably it is old news but I couldn't find it here that fast.
I have 20.000 photos, some have a keyword(s) some have not. So I searched for a selection of photographs without a keyword. But there is none.
The trick is to add a special keyword you normally not use like (only) "_" (underscore) to all photos without a keyword. But there is no way to select photos wihtout a keyword... So I came up with a two step strategy and this trick only cost 2-3 minutes with 20.000 photos:
Add the "_" keyword to all you photos.
Select all photos with a real keyword by shift clicking on all keywords tags except "_".
Remove the "_" keyword from those photos.
Now select the "_" keyword or search on the "_" keyword.
A how to, in 14 steps:
1. Hit G, goto library (in left panel) and click on "all photographs"
2. Select all photos by pressing ctrl-a
3. Wait till LR is ready displaying all used keywords in the right keywords panel
4. In right panel under keywording/keyword tags add the keyword "_"
5. Wait till LR is ready updating all your photos (20.000 photos: 1 minute)
6. Goto "Keyword tags" in the left panel
7. Select the (suppose the second) keyword just after the "_" keyword by left clicking.
8. Scroll down till the end in the keyword list
9. Now, shift click on the last keyword
10. You now selected all keywords except the "_" keyword and all are highlihted
11. Select all photo's (ctrl-a)
12. Wait till LR shows all the keywords in the Keywording/keyword tags panel on the right
check: the keyword "_" should have no asterix like "_*" if so: goto step 1!
13. Remove the "_" keyword and press enter
14. Wait till LR is ready
From now on while selecting the "_" keyword or searching on the keyword "_" LR will only show the photos without a (real) keyword.
Of course, as soon as you add keyword(s) to your foto(s) you have to delete the "_" keyword.
Speed tip: when starting keywording, select the photo you like to add real keywords, ctrl-k brings you directly to the keyword tags panel in edit mode, hit backspace (to delete the "_" keyword and start typing your keywords and hit enter or tab when done. Then immediately (of course) the photo dissapears from your "_" keyword selection photos and the next photo is selected and is waiting for you to hit ctrl-k and start again. Look mam, "no mouse used!"
Have fun!

>I have 20.000 photos, some have a keyword(s) some have not. So I searched for a selection of photographs without a keyword. But there is none.
There is actually, and it is much simpler:
Using the Find Panel, choose Text: Keywords and Rule: Are Empty.

SO HEADER TEXT

Dear Gurus,
When I create a new SO, header text will default one customer's text, it also pop up the message: Test is formatted -> Details in VA01.
However in screen Text determination analysis, these texts that get default content have access sequence, but I can't search text in SO10. (for example: Object: KNVV, ID: C003)
In this case, I wanna know the default text how to link to SO HEADER?
Thanks in advance.
Jamie

Hi Kiran,
Yes, I know how to search the access sequence.
After getting the text object & text ID, I can't find any relative text name with ID: C003 in SO10.
Or another factor impact the header text? thanks
Jamie

Sales order text is not updating

Hi All,
call function 'SD_SALESDOCUMENT_CHANGE'
    exporting
      salesdocument    = salesdocument
      order_header_inx = w_order_header_inx
      simulation       = testrun
      business_object = 'BUS2032'
      call_from_bapi   = 'X'
    tables
      return           = return
      sales_text       = order_text
      textlines_ex     = text_lines.
COMMIT WORK.
I am using above FM for updating Sales order test .I am Passing test in the Prameters      textlines_ex     = text_lines.
I am Unable to dig what the issue is .Can anyone help me out in this .
salesdocument = Sales Document number
Moderator Message: Even we cant help you, with a question like that. Put more effort into framing your questions.
Edited by: kishan P on Jan 9, 2011 6:56 PM

Sounds like a text determination config issue - have you assigned the text in VOTXN? Have you run the text determination analysis from VA02 in the document?

Keyword search on collection set not working for specific sub-collections

I am having the following problem:
When I do a search (Text->Keyword->Contains All-> for a keyword (in my
case "Fave"), it does not work if I select the collection set that contains the
collection that contains the images with that keyword.
It DOES work if I select the folder, parent folder, or just the
collection that contains the image. Just not the collection set.
The details of how this came about are below - and are probably a
little unusual, but not outside the realm of what should be allowed.
Is there a way to reset/reindex keywords in a collection?
Specifically:
images lives in collection C1 which is part of collection set
CS1. It lives in F1 (on disk and in LR) which is in parent folder
PF1. A keyword search works when I select F1 or PF1 or C1, but NOT
when I select CS1
This happens on a set of folders that went through the following
(totally allowable?) sequence:
I am running on a Windows 7 64 bit machine with 64 bit LR3.3
I have an internal disk where I keep my catalog and images. Windows
mounts this as P:. I also mount it on my Desktop as the folder
"Lightroom" (this allows me to change to a larger external disk
by mounting it in the same place without changing the locations of
all the files).
I mistakenly imported pics into P:/PF1 rather than
C:/.../Desktop/Lightroom/PF1 as I normally do.
Added keywords made a collection C1 put the images in it, and added
C1 to collection set CS1
Having realized my import mistake (P: rather than
C:/.../Desktop/Lightroom), I went in the Folders->P: section and did
an Update Folder Location on F1 to C:/.../Desktop/Lightroom/F1
This seemed to be all fine, but this weird Keyword search problem
resulted.
I have tried the following things:
o Deleting and re-adding Keywords in the Collection
o Deleting and re-adding the Keywords in the Folder
o Renaming and relocating the Folder
o Create new collection C2 and move pics from F1 into C2

See my simplified recipe for reproducing this bug (which turns out not to be related to the drive mounting I described):
http://forums.adobe.com/thread/764538

Filter for NOT [keyword]?

I cannot find any DIRECT way to Library Filter for photos NOT containing a particular "keyword." The work-around through Text/Keywords/Don't Contain doesn't work for me because many of my "keywords" contain spaces (multiple "words"). Since there doesn't seem to be any "quotation" option in the text-search box, what I get is a search that ORs together each "word" of my multi-word "keyword."
I've discovered that I can get around this problem by first making a Quick Collection from all of my positive filter criteria, then selecting the photos from that Quick Collection with the undesired (negative) "keyword" (for example, by clicking on one of my "keywords" in the Keyword List/Filter Keywords column), and finally removing these selections from the Quick Collection.
Alternatively, I can Rename my "keywords," replacing all spaces with hyphens, underscores, or whatever, and then use the Text/Keywords/Don't Contain filter.
There should be a more direct method. Am I missing something? -- JClarkW

I'm afraid you didn't understand the OP right.
He has 3 images with the following keywords:
aaa
bbb
"aaa bbb"
He wants to exclude #3 only. He doesn't want to exlude neighter #1 nor #2.
Your method will exclude:
!aaa - #1 and #3.
!bbb - #2 and #3.
!aaa !bbb - #1, #2 and #3.
So, it doesn't work. With the current search implementation it is impossible to achieve what the OP wants.

Text & Keyword analysis

Similar Messages

Maybe you are looking for