How to tokenize the unicode text into single word stream ?

Question

i am a newer to java, trying a toy program
e.g a% #unicode --> a unicode
ansi txt successeed
but unicode txt parse result �F a u n i c o u d e
how can i parse the unicode txt into single word by streamTokenizer?
thanks!
import java.io.*;
//import java.util.*;
public class Test
public static void main(String[] args)
try
BufferedReader r = new BufferedReader (new FileReader("ansiIn.dat"));
StreamTokenizer st = new StreamTokenizer(r);
st.lowerCaseMode(true);
st.whitespaceChars('\u0000', '\u0040');
st.whitespaceChars('\u005B', '\u0060');
st.whitespaceChars('\u007B', '\u00FF');
while(st.nextToken() != StreamTokenizer.TT_EOF)
String token = st.toString();
System.out.println(token);
catch(IOException exception)
exception.printStackTrace();
}

Answer 1

copy to clipboard
run this script
--[SCRIPT]
set leTexte to the clipboard as text
set leTexte to my remplace(leTexte, " :", ":")
set leTexte to my remplace(leTexte, space, tab)
set deci to character 2 of (0.5 as text)
set leTexte to my remplace(leTexte, ".", deci)
set the clipboard to leTexte
--=====
on remplace(t, d1, d2)
local l
set AppleScript's text item delimiters to d1
set l to text items of t
set AppleScript's text item delimiters to d2
set t to l as text
set AppleScript's text item delimiters to ""
return t
end remplace
--=====
--[/SCRIPT]
paste where you want.
The last call to the 'remplace' handler is used to replace the decimal period by the decimal char in use (here it is comma)
Yvan KOENIG (from FRANCE jeudi 9 avril 2009 12:25:19)