Tokenize input text in Java

Question

Let's say I have a text file that contains the following paragraphs:
"Please keep comments on topic.
Remember, the /dev/null forum can be used for test posting."
Now I want to put each token, including while space such as " " and newline, into a string. If I have a string tokens[], then
tokens[0] should have "Please";
tokens[1] should have " "(the white space following "Please" but before "keep"), if there is two white spaces between "Please" and "keep", I can put both the two white space into tokens[1], or put the latter one into tokens[2], not a problem;
tokens[2] should have "keep";
tokens[i] should have ".";
tokens[i+1] should have the newline.
tokens[i+2] should have "Remember".
Now the question is: Is there any easy way to do it? I don't think Sun provide any API to do that. Thanks in advance.

Answer 1

Yes, I have looked at the StringTokenizer class.
I am using the String.split method, which is similar
to StringTokenizer class' methods. However, other
while space, such as newline and tab, are not picked
up properly.
For example, "Software\nenginnering" will be treated
as one token.
I guess I will have to manually process the newline
and tab.Both the StringTokenizer class and the java.lang.String.split method will accept newline and tab delimiters; however, they work better if you read the instructions. Check this constructor "public StringTokenizer(String str)" and the java.util.regex.Pattern class API's.