Unicode word boundaries in Regex

Dear All,
I am using the following simple regex: "\\w+" which matches one more word characters. My problem is that it is not matching unicode characters which are part of my language. Is there any way to force java regex engine to include unicode characters in w ?
If this is not possible is there any third party library tha can be used?
Many thanks,
Chris Farrugia

On the javadoc page for java.util.regex.Pattern, it says the following:
Classes for Unicode blocks and categories
\p{InGreek} A character in the Greek block (simple block)
\p{Lu} An uppercase letter (simple category)
\p{Sc} A currency symbol
\P{InGreek} Any character except one in the Greek block (negation)
[ \ p { L } & & [ ^ \ p { L u } ] ] Any letter except an uppercase letter (subtraction) (Spaces added by me to prevent forum mungery.)
So you should be able to work something out using the Unicode support constructs.

Similar Messages

Determining word boundaries when no space exists in text

I am developing a text search feature for a viewer application and I run into PDFs quite often that do not use the space character to delineate word boundaries. For example, a text showing operator with individual glyph positioning will contain strings and positioning information like this:
[(de)15(grees)-262(and)-262(who)-262(w)10(ould)-262(contrib)20(ute)-262(an)]TJ
When the strings are concatenated the result is:
"degreesandwhowouldcontributean"
Without spaces it's not possile to split the string into words based on character information. It would appear the only information that could be used to guess word boundaries is the glyph positioning. I have tested the documents in Adobe Reader and the application is able to correctly determine where word boundaries are, and it must be doing so by examing the glyph positioning and metrics.
My first appreach was to get the glyph width for the space character, and assume a space is any position advance greater than the glyph width of a space. The problem with that is the case where the font has been subsetted and the 'space' glyph is missing from the font.
My second approach was to calculate the average glyph width for the font, then assume any text advance greater than 33% of the average glyph width is a space. Works better but still not a reliable general solution.
My question: does Adobe have a standard method for determining word boundaries when space characters are missing?

Sounds like it's time for me to play some more "guesswherethewhitespaceis". I've gleaned a bit by experimenting with Acrobat... feeding it text strings and increasing/decreasing glyph spacing to get an idea of how whitespace threshholds are being derived. I discovered Acrobat gets word boundaries wrong on occassion so it seems to be an inexact process at best.
Oh joy... in any case thanks for the reply.

Ignore word (Java.util.regex )

Hello All,
Can anyone help me to solve this probelm:
Probelm: I have a text file and i want to search a word or combination of words in that using java.util.regex
Example : in the sentence "things like the Forestry in the Commission (FC)." i want to search "Forestry Commission" while ignoring the word "in the". This ignoring criteria is specific i.e. search return true only if it ignore "in the" word not any other word.
Also how i ignore multiple words in ignore condition.
Thanks in advance.

Try out this line of code:
System.out.println("Forestry in the Commission".replaceAll("Forestry\\s(.*?\\s)?Commission", "Hello"));In EBNF, it looks like this:
match ::= "Forestry" <whitespace> [<character> <whitespace>] "Commission"
whitespace ::= <space> | \t | \n | \x0B | \f | \r
character ::= (any one character)
This, of course, is an ambiguous EBNF definition. The breakdown of the expression, however, reveals why this works. In the string, "\\s" refers to a character of whitespace. "(.*?\\s)?" is where the magic happens: it causes ".*?\\s" to happen either not at all or once. ".*?" will consume as few "." (any character) as possible to make the match, and the following "\\s" is to make sure that strings like "Forestry deCommission" don't match. The EBNF's ambiguity comes from EBNF's lack of ability to describe "reluctant qualifiers": qualifiers that indicate that as few of the given expression should be matched as possible.
Cheers!

Regex - match a word except when it's preceeded by another word

Does anyone know how to write a regular expression that will match an occurrence of a word except when it's preceeded by another word? I'm trying to match all occurrences of the word "function" except when it's part of the phrase "end function". Is that possible in a single regular expression?

Maybe this is just how it works, but I'm not sure why a string
with one space wouldn't match but a string with two would.At the beginning of the spaces, the lookbehind causes the match to fail, but then the Matcher bumps ahead one position and tries again. At that point, the lookbehind expression doesn't apply anymore, so you get a match. (You should be able to confirm this by counting the spaces in your output.) I tried using the "aggressive plus" to force it to treat all the spaces as one atom, but that didn't work:
Pattern p = Pattern.compile("(?<!end)(\\s++)function");I don't see how to do this using "pure" lookaround, but if you don't mind matching the preceding word, this will work:
Pattern p = Pattern.compile("(^|(?!end\\b)\\b\\w+ +)function\\b",
                              Pattern.MULTILINE);Getting pretty hairy, I know, but it matches the word "function", either as the first thing on the line, or preceded by a word that is not "end" (those first couple of \b's are there to ensure that only the whole word "end" will block the match). Here's how you would use this pattern to replace "function" with "method", except when it's preceded by "end":
import java.util.regex.*;
public class Test
public static void main(String[] args)
    String target = "end function\n"
                  + "function test\n"
                  + "functioning test\n"
                  + "test function\n"
                  + "test function end\n"
                  + "end    function\n"
                  + "ending function\n"
                  + "rend   function\n"
                  + "end   functioning\n";
    Pattern p = Pattern.compile("(^|(?!end\\b)\\b\\w+ +)function\\b",
                                Pattern.MULTILINE);
    Matcher m = p.matcher(target);
    target = m.replaceAll("$1method");
    System.out.println(target);
}Here's the output I get:
end function
method test
functioning test
test method
test method end
end    function
ending method
rend   method
end   functioningOf course, if you do know that there will always be exactly one space between "end" and "function", none of this is necessary; you can just use dcostakos's original lookbehind regex--except that I would add word boundaries:
Pattern p = Pattern.compile("(?<!end\\s)\\bfunction\\b");

Word wrapping at JTextPane boundaries

Hi ! I have two problems related to JTextPane -
1. I want to wrap words which are larger than the width of JTextPane so that it may remain visible inside the fixed boundaries.
2. if there is an image and it can not be displayed in the same line in continuation of text then it should be displayed in the next line.
All images that i have used have less width than that of JTextPane.
Both these problems may be related or different, I dont know how to solve these problems.
Please suggest any solution.

Till now I am able to find the height of JTextPane after inserting some text by creating a dummy JTextPane
and calling the method getPrefferedSize().height
this wraps text correctly at word boundaries but i want it to be wrapped according to word boundaries and
textpane boundaries both
actually I am developing a chat application in which I have to show message from other person
and if there is some text like ": )", ": D" etc then i want to show smileys :), :D in the place of these substrings
I can insert text and replace the text with images by using StyledDocument's insert function
but if i simply replace the text with images then the images are shown in the same line and wrapping doesn't work
hence some part of images are not visible in the JTextPane
as shown in the links http://www.facebook.com/photo.php?fbid=193222607456431&set=a.193222600789765.36992.100003060781951&type=1&ref=nf
and http://www.facebook.com/photo.php?fbid=193225164122842&set=a.193222600789765.36992.100003060781951&type=1&ref=nf
here link 1 shows the problem of partial display of image
and link 2 shows the problem of inserting text that is larger than the width of the JTextPane
And i want that whatever is inside the JTextPane should be shown inside the boundaries of it, I don't want to add a
horizontal scrollbar !!!!!
hope you got the problem, please provide some help !

A problem with regex and special characters

Hello,
I am using regex in my application but i have a problem with special characters. Here is the explanation of what i am doing:
I have a certain piece of text that i want to parse and replace every occurrence of a given word with some sort of a tag which have the word found inside it.
so that: go Going Go to gOschool by bus and to learn and to play GO Go
and i need to replace the word "go" (case insensitive and only at word boundaries) should be:
*<start>go<end> Going <start>Go<end> to gOschool by bus and to learn and to play <start>GO<end> <start>Go<end>*
Consider the following code and call the method with the parameter"go?"
The Matcher finds a weird match at the word "G?oing" with only the letter G !!!
It also ignores the "?" in the pattern completely.
Any clue of what is happening i would be very grateful...
private static String replaceMatches(String strToFind)
        String resultArticle="";
        String article = " "+"go? G?oing Go? to gOschool by bus and to learn and to play GO? Go?*"+" ";
        strToFind = "\\b"+ strToFind +"\\b";
        String linkPart1= "<start>";
        String linkPart2 = "<end>";
        Pattern p = null;
        try{
            p=Pattern.compile(strToFind, Pattern.CASE_INSENSITIVE);
        Matcher m = p.matcher(article);
        String[] res = p.split(article);
        int i=0;
        //System.out.println("result of split: "+res.length );
        while(m.find())
            resultArticle+=(res[i]+" ");
            resultArticle+=linkPart1;
            resultArticle+=m.group().trim();
            resultArticle+=(linkPart2+" ");
            i++;
        if(i<res.length)
            resultArticle+=res;
//System.out.println("result of match: " + i);
System.out.println(article);
//System.out.println(resultArticle.trim()+scripts);
catch(PatternSyntaxException ex){}
return resultArticle.trim();
}Thanks

tarek.mamdouh wrote:
because split will not work when trying to replace the first word if i don't append a space at the beginning.Split doesn't work anyway. And my question wasn't why do you add spaces (which you really don't need to do), but why do you do them with " " + "go" rather than just " go"
replaceAll will replace all the occurrences in the text with only one word. without taking into consideration the case of the word i need to replace.No.
>
If i use replacaAll(article, strToFind) the output will be:
<start>go?<end> G?oing <start>go?<end> to gOschool by bus and to learn and to play <start>go?<end> <start>go?<end>No. I showed you the actual output of an actual replaceAll.
which is not what i want as i need to keep the case of the words i am replacingThe replaceAll I showed you does that.
Please study the examples given and read the docs carefully rather than making claims based on inaccurate guesses.

Regular expression-- How to match whole word

I use java.util.regex in my case,
my problem is following:
here is a sentence: "oRs As ordoRs"
i use this pattern: (?!\w)oRs(?<!\w)
the first word oRs should match, but why the third word also match?
Maybe my problem also could be scribled as following:
How to match whole word?

here is a sentence: "oRs As ordoRs"
i use this pattern: (?!\w)oRs(?<!\w)
the first word oRs should match, but why the third
word also match?What's even more interesting, there should be no matches at all! Just because the pattern is inherently inconsistent:
the (?!\w) requires the next char to be a non-letter, but
at the same time this char is reqired to be a letter 'o'.
The same thing in the end: the last char should be 's' but not a letter. So the pattern must match nothing.
Maybe my problem also could be scribled as following:
How to match whole word?It's simple, use word boundaries - either directed ( "\<" and
"\>") or undirected ("\b").
The word pattern will look like "\b\w+\b" or "\<\w+\>"
Test it here
http://jregex.sourceforge.net/demoapp.html

Wrapping Words

Hi,
I am about to write a routine to output several paragraphs of text to a file. The file can have up to 80 characters on each line and I need to parser the text so that the file does not cut any words apart. JTextArea has setWrapStyleWord and setLineWrap to wrap lines at word boundaries rather than at character boundaries. To save me writing a routine that likely has already been written and is faster than what I work out can anyone point me in the right direction? I cannot find anything appropriate in the JDK.
Thanks in advance, Richard.

The word wrap in JTextArea does not wrap based on number of characters, it takes into account the size of words when rendered in the current font.
Word wrapping by number of characters is pretty easy. Just go to position 80, then back up until you hit a space. If you don't find any spaces on the line, then just wrap at the end of the line.

How to count words?

Hello, namless hero~
Just like the title said, if you has any idear, please reply this post, thanks!!

There are two basic ways I can think of to do this. Either get the text, and run your own word count alogorithm on it, or use the word boundary methods in ParagraphElement. In the first instance, you have more control over what is considered as a word, and in the second way you let the Player decide.
Option 1:
You can get the text of the TextFlow by exporting using the plain text filter:
     var text:String = TextFilter.export(textFlow, PLAIN_TEXT_FORMAT, ConversionType.STRING_TYPE) as String;
Once you have the text, just scan through it to find the words. This is the best way if you want to define for yourself what constitutes a word.
If there is going to be a lot of text, you may prefer to look at it paragraph by paragraph. So instead you would loop through the paragraphs in the flow, and call getText() on each one, and then run your algorithimn on the String returned by getText().
Option 2:
Otherwise, you could iterate through the paragraphs of the flow, looking for word boundaries. This will get you all possible word boundaries, including between spaces. To do this, it would look something like this:
     var paragraph:ParagraphElement = textFlow.getFirstLeaf().getParagraph();
     do {
          var relativePosition:int = paragraph.findWordBoundary(0);
          while (relativePosition < paragraph.textLength)
               trace("Word boundary at", paragraph.getAbsoluteStart() + relativePosition);
               relativePosition = paragraph.findWordBoundary(relativePosition);
          paragraph = paragraph.getNextParagraph();
     } while (paragraph != null);
I haven't compiled this or debugged it, and I coded it from memory, but hopefully this will serve as a guideline for what you could do.

[SOLVED]vim word wrap

Is it possible to set vim up to prevent the word wrap from splitting words. Like in more modern applications, the word wrap will try to move large words at the end of the screen to a new line if they are too big. Also, is this behavior possible in less?
Last edited by Bellum (2011-11-13 13:30:34)

Here's what I have to control line breaks:
set linebreak " Wrap breaks lines at word boundaries.
set nolist " Control TAB and EOL display
set textwidth=0 " Disable auto line breaks.
set wrapmargin=0 " Stop auto insertion of EOL.
set showbreak=>>\ \ " Indicate wrapped lines.
Last edited by thisoldman (2011-11-13 08:40:30)

Pattern-Matcher

hi guys.. i need help on pattern-matcher. below is a simplified portion of my problem.
using line A, if you run it.. u will see that when u type in *&word*, the result would be Doesn't Match. i believe it is due to the *\b* notation in the Pattern.compile where it specifies the word boundary that excludes whitespaces and punctuation mark.
I know it will work when i use line B instead. but the occurance would also be detected when i input blabla&word or *&wordblabla* or bla&wordbla (this is true but somehow it doesn't work in this simplified code)
thus.. my question is.. how do i specify a word boundary (like what \b does) and at the same time.. allow for punctuation marks to occur in my word?
for example.. i would like to encounter for these patterns:
*<word>*
*&word*
i hope u guys can help me out. thank you in advance.
import javax.swing.*;
import java.util.regex.*;
public class PatternMatcherTest
     public PatternMatcherTest()
          String regex = "&word";
          Pattern p = Pattern.compile("\\b" + regex + "\\b");   // -- line A
          //Pattern p = Pattern.compile(regex);                     // -- line B
          Matcher m = p.matcher(JOptionPane.showInputDialog(null, "Enter string: "));
          if (m.matches())
               System.out.println("Match!");
          else
               System.out.println("Doesn't Match");
     public static void main (String [] args)
          PatternMatcherTest pmt = new PatternMatcherTest();
}

I'm not sure I understand your requirements, but try this: Pattern p = Pattern.compile("(?<!\\w)" + regex + "(?!\\w)"); If that doesn't solve your problem, post some examples of inputs and the outputs that you want to get for them.
hlfrk414 wrote:
Have you tried making the word boundaries reluctant? If you want to see if the punctuation matches later, then change the \b to \b??
The ?? tell the regex to choose the minimum needed to make the group happy. So the \b?? wont take punctuation unless no other groups want it.Ummmm... no.

XQuery tokenize() regular expression not interpreting correctly?

I've been trying to use XQuery tokenize() with a regular expression as an argument.
Given the string
S831409 $50 070886
(Note: 3 spaces at the end.)
And the expression
\b(\w+)\b
to find words on word boundaries
and using java.util.regex functions Pattern() and Matcher(), this is returned:
S831409
50
070886
without leading or trailing spaces.
Using the XQuery:
depositAmt =
{snip}
return
let $fndWord := xf:tokenize($locVal, "\b(\w+)\b")
return $fndWord[2]
I would expect to get
depositAmt="50"
Instead I get
depositAmt=" $"
Okay, changing the return to
$fndWord[3]
I get
depositAmt=" "
Changing the return to
$fndWord
gets
depositAmt=" $ "
Changing the regEx to
\b\s+\b
improves things somewhat, but not much.
Am I missing something?

The pattern that is supplied to tokenize() function is used to indicate the SEPARATOR. So the pattern given to the function must match the separator instead of the content that you want. So you must think in the opposite.
For your case try to use [^\w]+ as the pattern. I think it should give you the same result.
Good luck

Question on the parameter type of DLL functions

Question on the parameter type of DLL functions
I am trying to develop an interface to control a laser with LV8.2 I am planing to use the VI “Call Library Function Node” to call the DLL and set the exact same parameters of functions in the DLL.
According to the DLL manual of the laser, there are six functions. One of the prototypes is: function getstatuspointer : pointer. I don't know how to set this parameter for the function. Is there anyone who can give some hints? Thanks!
BTW :The following is a description of the function:
This function returns a 32 bit pointer to the STATUS data structure (Tstatusrec). In the 32 bit DLL, every application is using its own copy of the data structure (local
memory). The data structure is a packed structure. "packed" means that the fields in the structure are not aligned on word or double-word boundaries.
Tstatusrec = packed record
                        size : word;
                        initstat : byte;
                        queuefill : byte;
                        anzapp16 : word; (not used)
                        anzapp32 : word;
                        anzdock16 : word; (not used)
                        anzdock32 : word;
                        dll16ver : tchararray; (not used)
                        dll32ver : tchararray;
                        excimerver : tchararray;
                        excimer : TExcimerStatus;
                  end;

Hi Jack,
This is a good starting point when using Call Library Function node. Check out this Help Page
Van L
NI Applications Engineer

Regular expressions in Format Definition add-on

Hello experts,
I have a question about regular expressions. I am a newbie in regular expressions and I could use some help on this one. I tried some 6 hours, but I can't get solve it myself.
Summary of my problem:
In SAP Business One (patch level 42) it is possible to use bank statement processing. A file (full of regular expressions) is to be selected, so it can match certain criteria to the bank statement file. The bank statement file consists of a certain pattern (look at the attached code snippet).
:61:071222D208,00N026
:86:P 12345678BELASTINGDIENST       F8R03782497                $GH
$0000009                         BETALINGSKENM. 123456789123456
0 1234567891234560
:61:071225C758,70N078
:86:0116664495 REGULA B.V. HELPMESTRAAT 243 B 5371 AM HARDCITY HARD
CITY 48772-54314
:61:071225C425,05N078
:86:0329883585 J. MANSSHOT PATTRIOTISLAND 38 1996 PT HELMEN BIJBETA
LING VOOR RELOOP RMP1 SET ORDERNR* 69866 / SPOEDIG LEVEREN
:61:071225C850,00N078
:86:0105327212 POSE TELEFOONSTRAAT 43 6448 SL S-ROTTERDAM MIJN OR
DERNR. 53846 REF. MAIL 21-02
- I am in search of the right type of regular expression that is used by the Format Definition add-on (javascript, .NET, perl, JAVA, python, etc.)
Besides that I need the regular expressions below, so the Format Definition will match the right lines from my bankfile.
- a regular expression that selects lines starting with :61: and line :86: including next lines (if available), so in fact it has to select everything from :86: till :61: again.
- a regular expression that selects the bank account number (position 5-14) from lines starting with :86:
- a regular expression that selects all other info from lines starting with :86: (and following if any), so all positions that follow after the bank account number
I am looking forward to the right solutions, I can give more info if you need any.

Hello Hendri,
Q1:I am in search of the right type of regular expression that is used by the Format Definition add-on (javascript, .NET, perl, JAVA, pythonetc.)
Answer: Format Definition uses .Net regular expression.
You may refer the following examples. If necessary, I can send you a guide about how to use regular expression in Format Defnition. Thanks.
Example 6
Description:
To match a field with an optional field in front. For example, u201C:61:0711211121C216,08N051NONREFu201D or u201C:61:071121C216,08N051NONREFu201D, which comprises of a record identification u201C:61:u201D, a date in the form of YYMMDD, anther optional date MMDD, one or two characters to signify the direction of money flow, a numeric amount value and some other information. The target to be matched is the numeric amount value.
Regular expression:
(?<=:61:\d(\d)?[a-zA-Z]{1,2})((\d(,\d*)?)|(,\d))
Text:
:61:0711211121C216,08N051NONREF
Matches:
1
Tips:
1.     All the fields in front of the target field are described in the look behind assertion embraced by (?<= and ). Especially, the optional field is embraced by parentheses and then a u201C?u201D (question mark). The sub expression for amount is copied from example 1. You can compose your own regular expression for such cases in the form of (?<=REGEX_FOR_FIELDS_IN_FRONT)(REGEX_FOR_TARGET_FIELD), in which REGEX_FOR_FIELDS_IN_FRONT and REGEX_FOR_TARGET_FIELD are respectively the regular expression for the fields in front and the target field. Keep the parentheses therein.
Example 7
Description:
Find all numbers in the free text description, which are possibly document identifications, e.g. for invoices
Regular expression:
(?<=\b)(?<!\.)\d+(?=\b)(?!\.)
Text:
:86:GIRO 6890316
ENERGETICA NATURA BENELU
AFRIKAWEG 14
HULST
3187-A1176
TRANSACTIEDATUM* 03-07-2007
Matches:
6
Tips:
1.     The regular expression given finds all digits between word boundaries except those with a prior dot or following dot; u201C.u201D (dot) is escaped as \.
2.     It may find out some inaccurate matches, like the date in text. If you want to exclude u201C-u201D (hyphen) as prior or following character, resemble the case for u201C.u201D (dot), the regular expression becomes (?<=\b)(?<!\.)(?<!-)\d+(?=\b)(?!\.)(?!-). The matches will be:
:86:GIRO 6890316
ENERGETICA NATURA BENELU
AFRIKAWEG 14
HULST
3187-A1176
TRANSACTIEDATUM* 03-07-2007
You may lose some real values like u201C3187u201D before the u201C-u201D.
Example 8
Description:
Find BP account number in 9 digits with a prior u201CPu201D or u201C0u201D in the first position of free text description
Regular expression:
(?<=^(P|0))\d
Text:
0000006681 FORTIS ASR BETALINGSCENTRUM BV
Matches:
1
Tips:
1.     Use positive look behind assertion (?<=PRIOR_KEYWORD) to express the prior keyword.
2.     u201C^u201D stands for that match starts from the beginning of the text. If the text includes the record identification, you may include it also in the look behind assertion. For example,
:86:0000006681 FORTIS ASR BETALINGSCENTRUM BV
The regular expression becomes
(?<=:86:(P|0))\d
Example 9
Description:
Following example 8, to find the possible BP name after BP account number, which is composed of letter, dot or space.
Regular expression:
(?<=^(P|0)\d)[a-zA-Z. ]*
Text:
0000006681 FORTIS ASR BETALINGSCENTRUM BV
Matches:
1
Tips:
1.     In this case, put BP account number regular expression into the look behind assertion.
Example 10
Description:
Find the possible document identifications in a sub-record of :86: record. Sub-record is like u201C?00u201D, u201C?10u201D etc. A possible document identification sub-record is made up of the following parts:
u2022     keyword u201CREu201D, u201CRGu201D, u201CRu201D, u201CINVu201D, u201CNRu201D, u201CNOu201D, u201CRECHNu201D or u201CRECHNUNGu201D, and
u2022     an optional group made up of following:
     a separator of either a dot, hyphen or slash, and
     an optional space, and
     an optional string starting with keyword u201CNRu201D or u201CNOu201D followed by a separator of either a dot, hyphen or slash, and
     an optional space
u2022     and finally document identification in digits
Regular expression:
(?<=\?\d(RE|RG|R|INV|NR|NO|RECHN|RECHNUNG)((\.|-|/)\s?((NR|NO)(\.|-|/))?\s?)?)\d+
Kind Regards
-Yatsea

Using jtextpane as jlist cell renderer component

hi,
I want to use Jlist (in a Jscrollpane) to list a series of boxes of text. The boxes have to be kept the same width, but the height can vary depending on the amount of text.
I want to use jtextpane because it wraps automatically on word boundaries... although I am confused by the jtextpane functionality...
but I just can't seem to crack it: presumably its going to involve
class MyCellRenderer extends JTextPane implements CellRenderer {
public Component getListCellRendererComponent( ...
then what ??? help!
mike rodent
PS also, how to make Jlist put a line (a single line) between each of the components in its list... it's no good doing setBorder inside the above method, as you then get 2 lines coalescing between adjacent Jlist elements...

PS also, how to make Jlist put a line (a single line) between each of
the components in its list... it's no good doing setBorder inside the
above method, as you then get 2 lines coalescing between adjacent
Jlist elements...Who says you need to have a Border with top and bottom lines?

Unicode word boundaries in Regex

Similar Messages

Maybe you are looking for